Ground truth

What ground truth is

Ground truth is a collection of expected or reference values that reference-based evals compare model outputs against. Instead of measuring subjective qualities like tone or safety, a reference-based eval checks whether the generated output matches the known-correct answer. You upload ground truth as a table of rows, attach it to specific eval templates, and those templates compare each model output to its reference at eval time.

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
  G["Ground truth<br/>inputs + correct outputs"] --> M["Mapped to a template<br/>variables + roles"]
  M --> C["Compare<br/>output vs the reference"]
  M --> F["Few-shot<br/>similar rows shown to the evaluator"]
  C --> S["Score"]
  F --> S

Read it left to right: one upload, mapped once, working two jobs. The reference gives comparison evals their answer key, and the same rows calibrate the evaluator on judged runs.

When to use ground truth

Ground truth is the right choice when you have a clear, correct answer to compare against. Common cases:

• A question answering system where the correct answer is known in advance
• A code generation tool where you know what the correct code should produce
• A data extraction pipeline where the expected fields and values are labeled
• A classification task where the correct label is on file
• Any task where correct means matching a specific reference output

If you do not have known-correct answers yet, or correctness is subjective like tone or helpfulness, let an evaluator model judge the response directly instead.

Evals that use ground truth

Several built-in templates compare against a reference value:

• Ground Truth Match: pass or fail on whether the output matches the reference value semantically
• Answer Similarity: a similarity score between the output and a reference answer
• Fuzzy Match: approximate string matching with configurable tolerance
• Retrieval metrics such as Recall@K and Mean Average Precision: measure whether a retrieved set contains the expected items

Each template documents the input keys it expects. When you attach ground truth to a template, you map your columns to those keys.

How ground truth data is structured

You upload ground truth as a CSV, XLS, XLSX, or JSON file; the platform stores it as rows with named columns. This is its own upload attached to a template, not a dataset in the Datasets product. At minimum it needs:

• Input columns: the data the model receives, such as the question column for a question-answering eval
• Output column: the correct answer the eval compares against. Every row must have one
• Optional explanation column: the reasoning behind the answer, shown to the evaluator alongside it

Mapping ground truth to eval inputs

When you attach ground truth to an eval template, you configure two mappings:

• Variable mapping connects each eval template input to a ground truth column. If the template needs a question input, you map it to your prompt column. At eval time the platform uses this mapping to pull the right column value for each row, and it is also what similarity retrieval matches on
• Role mapping labels the semantic parts: which column is the correct output (required), and which column carries the explanation behind it (optional)

Retrieval and few-shot injection

Ground truth can also calibrate the evaluator by injecting similar examples into its prompt, called few-shot grounding. When enabled, the platform:

1. Embeds your ground truth rows so similar examples can be found
2. At eval runtime, finds the rows most similar to the current input (three by default)
3. Includes those rows in the evaluator prompt as reference calibration

This helps the evaluator apply your correctness criteria without you rewriting the prompt.

The platform embeds ground truth asynchronously after you save a mapping, and retrieval starts on the next eval run once embedding completes. If you change the variable mapping, the embeddings are marked stale and re-embedded automatically.

Keep exploring