Trace error analysis
The four per-trace quality scores and why they stay out of the feed
What Error Analysis is
Open a trace that has one and you’ll find Error Analysis in the Scores accordion: four quality dimensions, each scored for that one trace. It’s a separate, per-trace view, not a property of an issue or a cluster.
The four dimensions
- Factual Grounding: whether the response holds up against the evidence and context the agent actually had
- Privacy And Safety: whether the response handles sensitive data and follows safe practices
- Instruction Adherence: whether the response follows the instructions the agent was given
- Optimal Plan Execution: whether the agent’s sequence of decisions and tool calls was the right one for the task
Note
The UI title-cases the raw dimension name, so what you’d write as “Privacy & Safety” renders as Privacy And Safety in the product.
Where you see it
Open the trace detail drawer and its Scores accordion shows one chip per dimension: the label and the score out of 5, for example “Factual Grounding 4/5”.
When to check the scores
Read the scores when you’re already looking at a specific trace, for example while working through an issue in the Investigate an issue guide, and want a read on that trace beyond the issue’s category. Treat a dimension scoring lower than the others as a pointer to look closer at the plan, the tool calls, or the response, not a verdict on its own.
A different pipeline from the Error Feed scanner
flowchart TD accTitle: Two independent reads on one trace accDescr: A single trace is read by two separate pipelines. The Error Feed scanner reads the trace and writes findings that group into issues in the feed. Error Analysis reads the same trace and writes four dimension scores into that trace's Scores accordion. TR["Trace"] -->|Error Feed scanner| FD["Finding → issue in the feed"] TR -->|Error Analysis| SC["Four dimension scores → Scores accordion"]
Error Analysis and the Error Feed scanner are two independent reads on the same trace. The feed reads traces, groups the problems it finds into issues, and tells you where to fix them; the Scores accordion tells you how one specific trace performed on these four dimensions.
These four scores don’t feed the error taxonomy: a low score doesn’t create an issue, and it isn’t a value you can filter the feed by. Checking both means opening the trace and reading the accordion yourself.
Keep exploring
Questions & Discussion