Error taxonomy

The shape of a finding

Every scanner finding Error Feed writes carries three tags: a group, a category inside that group, and a fix layer (the part of your system the finding points at). Eval-sourced findings are tagged differently: the eval name stands in for group, and category is left unset. Fix layer is the only one of the three that’s a live filter on the feed, with options for All Fix Layers, Prompt, Tools, Orchestration, and Guardrails. See Issue fields & filters for the full list of fields and filters.

The set is fixed, not free-form. A scanner finding always lands in exactly one row of the table below, and points at one of four fix layers.

flowchart TD
accTitle: How the five groups map onto four fix layers
accDescr: Tool Failures maps to Tools. Context and Retrieval maps to Prompt. Planning and Goals maps to Orchestration. Output Quality also maps to Prompt, which is why five groups resolve to only four fix layers. Infrastructure maps to Guardrails.
A["Tool Failures<br/>5 categories"] --> T["Tools"]
B["Context & Retrieval<br/>4 categories"] --> P["Prompt"]
D["Output Quality<br/>3 categories"] --> P
C["Planning & Goals<br/>3 categories"] --> O["Orchestration"]
E["Infrastructure<br/>5 categories"] --> G["Guardrails"]

Fix layers

• Tools: fix layer for Tool Failures findings
• Prompt: fix layer for Context & Retrieval and Output Quality findings
• Orchestration: fix layer for Planning & Goals findings
• Guardrails: fix layer for Infrastructure findings

Groups, categories & fix layers

Five groups organize twenty categories.

Group	Category	What it means	Fix layer
Tool Failures	Tool-related	The agent mishandled a tool or its result: asserted success after an error, passed wrong arguments, or guessed instead of calling	Tools
Tool Failures	Tool Selection Errors	The agent picked the wrong tool, or no tool, for the task	Tools
Tool Failures	Tool Output Misinterpretation	The agent misread or misused the result a tool returned	Tools
Tool Failures	Formatting Errors	The agent’s tool call or output didn’t match the expected format	Tools
Tool Failures	Language-only	A hallucination purely in language, no tool involved	Tools
Context & Retrieval	Context Handling Failures	The agent lost, dropped, or mishandled context it was given	Prompt
Context & Retrieval	Poor Information Retrieval	The agent retrieved information that was irrelevant, incomplete, or wrong	Prompt
Context & Retrieval	Incorrect Memory Usage	The agent used stored memory incorrectly, including outdated or unrelated memory	Prompt
Context & Retrieval	Unsupported Claim	The agent stated something that tool output or the end user’s own input doesn’t support	Prompt
Planning & Goals	Task Orchestration	The agent sequenced or delegated steps incorrectly	Orchestration
Planning & Goals	Goal Deviation	The agent drifted from the goal it was given	Orchestration
Planning & Goals	Resource Abuse	The agent used excessive steps, calls, or resources to complete the task	Orchestration
Output Quality	Instruction Non-compliance	The agent’s output didn’t follow the instructions it was given	Prompt
Output Quality	Incorrect Problem Identification	The agent misunderstood or misidentified the problem it was asked to solve	Prompt
Output Quality	Incomplete Response	The agent’s response was absent, empty, or truncated	Prompt
Infrastructure	Environment Setup Errors	The agent’s runtime environment wasn’t configured correctly	Guardrails
Infrastructure	Resource Not Found	The agent tried to reach a resource that doesn’t exist	Guardrails
Infrastructure	Authentication Errors	The agent failed to authenticate with a required service	Guardrails
Infrastructure	Timeout Issues	A call the agent depended on didn’t complete in time	Guardrails
Infrastructure	Service Errors	A service the agent depended on returned an error	Guardrails

Keep exploring