Add Eval Configs to Run Test

Add one or more evaluation configurations to an existing test run. Specify template, name, config mapping, model, and error localizer per config. Returns created eval config objects.

POST https://api.futureagi.com/simulate/run-tests/{run_test_id}/eval-configs/

Authentication

X-Api-Key API Key Required

Your Future AGI API key used to authenticate requests. You can find and manage your API keys in the Dashboard under Settings.

X-Secret-Key Secret Key Required

Your Future AGI secret key, used alongside the API key for request authentication. This is generated when you create an API key in the Dashboard.

Path parameters

run_test_id UUID Required

UUID of the test run to add evaluation configurations to.

Request body

evaluations_config array of objects Required

Array of evaluation configuration objects. Each object supports the following fields:

  • template_id (string, UUID, required) — UUID of the evaluation template to use.

  • name (string, optional) — Name for this evaluation configuration. Defaults to Eval-<template_id> if omitted. Must be unique within the test run.

  • config (object, optional) — Template-specific configuration parameters.

  • mapping (object, optional) — Maps test execution data fields to the evaluation template’s expected inputs.

  • filters (object, optional) — Filter criteria to restrict which test results are evaluated.

  • error_localizer (boolean, optional) — Enables granular error localization on evaluation failures. Defaults to false.

  • model (string, optional) — Model to use for running this evaluation.

Response

201 Created
message string
Confirmation message indicating how many evaluation configs were added.
created_eval_configs array of objects

Array of created evaluation configuration objects. Each object contains: id, name, config, mapping, filters, error_localizer, model, status, eval_group, and template_id.

run_test_id string
UUID of the parent test run.
warnings array of strings
Non-fatal issues encountered while processing individual configs. Only present if partial failures occurred.

Errors

400 Bad Request Optional

Validation error. Common causes: empty evaluations_config, duplicate name within request, name already exists in test run, non-existent template_id.

{
  "evaluations_config": ["Duplicate eval name 'My Eval Config' found in the request. Each evaluation config must have a unique name."]
}

Or for existing name conflict:

{"error": "An evaluation config with the name 'My Eval Config' already exists in this run test. Please use a different name."}
401 Unauthorized Optional

Missing or invalid X-Api-Key or X-Secret-Key headers.

404 Not Found Optional

No test run found with the specified run_test_id.

{"detail": "No RunTest matches the given query."}
500 Internal Server Error Optional

Unexpected server error.

{"error": "Failed to add evaluation configs: <message>"}
GET /
Authentication
REQUEST 
 
 
  
 
 RESPONSE