Error localization

Which input caused the failure

When an evaluation fails, error localization narrows down the cause by analyzing which input field led to the failure, using the evaluator model’s explanation and the eval template’s criteria. Instead of knowing only that an eval failed, you learn which input, whether the prompt, context, or query, was responsible, so you can see whether the problem is in your data, your retrieval, or your instructions. This works the same whether you’re testing in the Playground or reviewing a live trace in Observe. You turn it on with the Error Localization checkbox when you configure an eval, on the template or a single run; see Evaluate via Platform & SDK for the flow.

How it works

1. Evaluation fails: error localization runs when a result comes back failed, pass/fail evals by their own verdict, scored evals on a low score.
2. Analysis: an error-localizing agent receives the input data, the eval criteria, the eval result, and the evaluator model’s explanation, then works out which input drove the failure.
3. Result: you get back an error_analysis block describing the problem and a selected_input_key naming the input field identified as the issue.
4. Review: the flagged field shows up in the trace detail eval view in Observe, and in the Playground’s run tests.

When it runs

Error localization runs on a failed result, as long as the eval isn’t a code-type or composite eval. It’s skipped when:

• Evaluation passed, nothing to localize
• Code-type eval, not supported
• Composite eval, not supported yet

Where results appear

• Trace detail eval view: looking at a span’s eval result in Observe shows the flagged input and analysis inline
• Develop’s dataset row drawer: open a row on a dataset run to see the same breakdown
• Playground: the Dataset, Trace, and Simulation test modes render it while you’re testing a template
• SDK: the result object returns error_analysis and selected_input_key directly

Keep exploring