Feedback
Record corrections and human signals on eval results to improve your evaluator over time
What feedback is
Feedback is a human or system signal recorded on an evaluation result, marking whether you agree with the judgment. When an eval returns a wrong result, you submit feedback with the correct verdict and an optional explanation. That feedback is stored and used to improve future runs: it steers the evaluator model with examples of what you called right and wrong, and it powers the correction loop that teaches custom evals your domain’s definition of quality.
Feedback turns scattered disagreements into systematic improvement, so evaluators converge on what your team means by correct.
How feedback is captured
Each feedback entry records:
| Field | What it holds |
|---|---|
| Value | Your verdict: passed or failed, what the eval should have returned |
| Explanation | Why you are correcting the eval, optional, up to 5000 characters |
| Improvement suggestion | How the eval could be better, optional, up to 5000 characters |
| Source | Where the eval ran: dataset, trace, experiment, eval_playground, prompt, observe, or sdk |
| Source ID | The row, trace, or item the feedback refers to |
Every entry is tied to the eval template, the person who submitted it, and the evaluated row, so you can pull up every correction for a given eval. On a dataset eval you also pick what happens next when you submit: retune the eval with your correction, re-run just that row, or re-run the whole dataset.
How feedback improves the next run
Corrections don’t sit in a log. When you submit feedback on a dataset eval, the platform stores the corrected row against the eval template. Every later run of that eval retrieves the most similar past corrections and hands them to the evaluator model as few-shot examples, so it sees what you called right and wrong before it scores.
%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
A["You correct<br/>a result"] --> B["Correction stored<br/>against the template"]
B --> C["Next run retrieves<br/>similar corrections"]
C --> D["Evaluator model scores<br/>with your examples"]
The template itself never changes: the criteria and version stay put. The steering happens at run time, each time the eval runs.
The correction loop
The correction loop is the pattern that turns feedback into a better eval: run an eval, find rows where it disagrees with your judgment, record the corrections, rewrite the eval rules to include those corrections as examples, then re-run and confirm agreement climbs.
- Establish a baseline: run a built-in eval on a batch of rows and compare its results to your manual verdicts.
- Find disagreements: a disagreement is any row where the eval and your team reach different verdicts. These rows teach the evaluator something new.
- Encode corrections: write a custom eval whose rule prompt states your domain rules and includes a few disagreement rows as few-shot examples for the evaluator model.
- Re-score and measure: run the new eval on the same batch and measure agreement against your verdicts.
- Iterate: pull a fresh batch, add new examples, and increment the version. Most evals converge in a few rounds.
See the eval correction loop cookbook for a worked end-to-end example.
Sources
Feedback can come from any surface where you evaluate, so you can close the loop wherever the eval ran:
| Source | Comes from | Use case |
|---|---|---|
| Dataset | Dataset column results | Labeling rows where bulk evals got it wrong |
| Traces | Live traces in Observe | Marking production responses that were misscored |
| Experiment | Experiment results | Marking experiment outcomes that did not match expectations |
| Eval playground | The eval tester | Quick feedback on an eval without running it on data |
| Prompt workbench | Prompt run results | Correcting scores while iterating on a prompt |
| SDK | Programmatic collection | Collecting corrections at scale in scripts or notebooks |
Keep exploring
Questions & Discussion