Create Custom Eval Template — API

Create a reusable custom eval template with criteria, output type (e.g. Pass/Fail), required keys, and model configuration.

POST https://api.futureagi.com/model-hub/create_custom_evals/

Authentication

X-Api-Key API Key Required

Your Future AGI API key used to authenticate requests. You can find and manage your API keys in the Dashboard under Settings.

X-Secret-Key Secret Key Required

Your Future AGI secret key, used alongside the API key for request authentication. This is generated when you create an API key in the Dashboard.

Request body

name string Required
Name for the eval template. Maximum 255 characters.
description string Optional

Description of what the eval template assesses.

criteria string Optional

Evaluation criteria text used by the LLM evaluator. Maximum 100,000 characters. Required unless config.data_injection is enabled — when data injection is active the evaluator runs directly on injected data without needing template variables in the criteria.

output_type string Optional

Type of evaluation output produced by the template. Accepted values: Pass/Fail, score, choices. Defaults to Pass/Fail.

required_keys array Required

Array of template variable names that must be mapped to dataset columns.

config object Optional

Configuration object controlling evaluation behavior and visibility.

model string

Model to use for running evaluations. Default is turing_small.

proxy_agi boolean

Whether to route evaluation requests through the AGI proxy. Default is true.

visible_ui boolean

Whether the eval template is visible in the dashboard UI. Default is true.

reverse_output boolean

Whether to invert the output logic of the evaluation.

config object

Additional template-level configuration parameters (e.g. LengthBetween bounds).

tags array Optional

Array of tag strings for categorizing the eval template.

choices object Optional

Key-value mapping of choice options. Required when output_type is choices.

check_internet boolean Optional

Whether the evaluation should check internet sources during assessment.

multi_choice boolean Optional

Whether multiple choices can be selected when output_type is choices.

template_id string Optional

UUID of an existing eval template to use as a base.

template_type string Optional

Internal template classification. Defaults to futureagi. Pass only when creating templates with a specific provider type.

Response

200 OK
data object

Response payload containing the new eval template identifier.

eval_template_id string
UUID of the newly created eval template.
success boolean
Whether the request completed successfully.

Errors

400 Bad Request Optional
Invalid request parameters.
401 Unauthorized Optional
Invalid or missing API credentials.
500 Internal Server Error Optional
Unexpected server error.
GET /
Authentication
REQUEST 
 
 
  
 
 RESPONSE