Evaluator models

What an evaluator model does

An evaluator model reads a response and applies an eval template’s criteria to produce a result, and an optional reason. It receives the text to evaluate, the template’s rule, and the required inputs, then returns a verdict. An evaluator model only reads and scores; it never generates or edits your AI’s output. Agent Evaluator and LLM-as-Judge evals use an evaluator model; Code Evals do not.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
  IN["Response + template criteria"] --> J["Evaluator model<br/>reads and scores"]
  J --> OUT["Result + optional reason"]

Future AGI evaluator models

Future AGI ships proprietary models built for evaluation, not for general-purpose chat:

Model	Code	Inputs	Best for	Latency
TURING_LARGE	turing_large	Text, image, audio	Max accuracy, multimodal evals	Higher
TURING_SMALL	turing_small	Text, image	High fidelity at lower cost	Medium
TURING_FLASH	turing_flash	Text, image	Fast, high-accuracy evals	Low
PROTECT	protect	Text, audio	Safety, guardrails, custom rules	Low
PROTECT_FLASH	protect_flash	Text	First-pass binary filtering	Ultra-low

Bring your own LLM

You can also use your own model as the evaluator, from OpenAI, Bedrock, SageMaker, Vertex AI, Azure, or a custom endpoint. Reach for a custom model when you need a domain-tuned model, must keep inference in a region, or already pay for a model you want to reuse. See Use Custom Models.

Modality

An evaluator model can only score inputs it can read. Image evals run on any of the Turing models; audio evals need turing_large or protect. A text-only model cannot score an image or an audio input, so pick a multimodal one when the eval carries one.

Keep exploring