Overview

Measure how well your AI achieves the user's goal

Evaluation is Future AGI’s quality measurement layer: it scores every response your AI produces against a definition of “good” that you control. And it’s audio native, so voice agents get scored as directly as text ones.

What is evaluation?

An LLM’s output is free text, and whether it is right is a judgment. Evaluation turns that judgment into a measurement. You define what “good” means once, as an eval, and the platform applies it to every response automatically, the same way every time.

Each eval scores one goal on one metric, for example:

  • Task completion: did the response do what was asked
  • Factual accuracy: are its claims true to the source
  • Safety: is it free of toxicity, prompt injection, and data leaks
  • Tone: does it speak the way your product should

Every run returns a score (pass/fail, a number, or a category) and, when an evaluator model is involved, a plain-language reason. Because the definition is fixed, the same bar is applied to every response: scores stay comparable across a dataset, a live trace, or a pull request, and a threshold on that score becomes a decision you can automate instead of a vibe check.

The quality loop

Evaluation is the middle of a loop, not a standalone tool:

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
  O["Observe<br/>what the agent did"] --> E["Evaluate<br/>score it against your bar"]
  E --> OPT["Optimize<br/>the prompt or model"]
  OPT --> G["Enforce<br/>gate the release"]
  G --> O
  style E fill:#2f2f2f,stroke:#ffffff,stroke-width:2px
  • Observe records what your agent did, and its traces become eval inputs
  • Evaluate scores each response and shows the result back on the trace
  • Optimization consumes the scores to improve the prompt or model
  • A threshold turns the score into a merge gate in CI/CD before anything ships

Start here

Evaluate via Platform & SDK

Every surface you can run an eval on, and how to get going

Built-in evals

156 ready evaluators across quality, safety, RAG, format, and more

Understand the model

Four objects carry the whole product:

  • A template defines what to measure
  • A config points it at your data
  • A run executes
  • A score comes out

A few short pages give you the whole mental model:

Understanding Evaluation

The template, config, run, and score model, and the quality loop

Eval types

Agent Evaluator, LLM-as-Judge, and Code Eval, and when to reach for each

Evaluator models

What decides the score, and how to choose one

Output types & scoring

The value, optional reason, and aggregates you get back

How it connects

Beyond the loop, the same evals run offline: datasets score rows in bulk, and simulations score synthetic conversations before anything reaches production.

Score traces in Observe

Attach an eval to a project’s traces and score new data as it arrives

Evaluate CI/CD Pipeline

Turn an eval threshold into a merge gate that blocks regressions
Was this page helpful?

Questions & Discussion