Create Custom Eval Template

Create a custom evaluation template with criteria, output type, and model configuration.

POST https://api.futureagi.com/model-hub/create_custom_evals/

Authentication

X-Api-Key API Key Required

Your Future AGI API key used to authenticate requests. You can find and manage your API keys in the Dashboard under Settings.

X-Secret-Key Secret Key Required

Your Future AGI secret key, used alongside the API key for request authentication. This is generated when you create an API key in the Dashboard.

Request body

name string Required
Name for the eval template. Maximum 255 characters.
description string Optional

Description of what the eval template assesses.

criteria string Required
Evaluation criteria text used by the LLM evaluator. Maximum 100,000 characters.
output_type string Optional

Type of evaluation output produced by the template.

required_keys array Required

Array of template variable names that must be mapped to dataset columns.

config object Optional

Configuration object controlling evaluation behavior and visibility.

model string

Model to use for running evaluations. Default is turing_small.

proxy_agi boolean

Whether to route evaluation requests through the AGI proxy. Default is true.

visible_ui boolean

Whether the eval template is visible in the dashboard UI. Default is true.

reverse_output boolean

Whether to invert the output logic of the evaluation.

tags array Optional

Array of tag strings for categorizing the eval template.

choices object Optional

Key-value mapping of choice options. Required when output_type is choices.

check_internet boolean Optional

Whether the evaluation should check internet sources during assessment.

multi_choice boolean Optional

Whether multiple choices can be selected when output_type is choices.

template_id string Optional

UUID of an existing eval template to use as a base.

Response

200 OK
data object
Response payload containing the new eval template identifier.
eval_template_id string
UUID of the newly created eval template.
success boolean
Whether the request completed successfully.

Errors

400 Bad Request Optional
Invalid request parameters.
401 Unauthorized Optional
Invalid or missing API credentials.
500 Internal Server Error Optional
An unexpected error occurred on the server while creating the eval template.
GET /
Authentication
REQUEST 
 
 
  
 
 RESPONSE