Thinking & Response Issues
Mistakes in understanding, reasoning, factual grounding, or output formatting.| Subcategory | Error Type | Description |
|---|---|---|
| Hallucination Errors | Hallucinated Content | Output includes information that is invented or not supported by input data. |
| Ungrounded Summary | Summary includes claims not found in the retrieved chunks or original context. | |
| Information Processing | Poor Chunk Match | Retrieved irrelevant or unrelated context. |
| Wrong Chunk Used | Response based on wrong part of retrieved content. | |
| Tool Output Misinterpretation | Misread or misunderstood the output returned by a tool or API. | |
| Decision Errors | Wrong Intent | Misunderstood the core user goal or instruction. |
| Tool Misuse | Used a tool incorrectly or in the wrong context. | |
| Wrong Tool Chosen | Selected an inappropriate tool for the task. | |
| Invalid Tool Params | Passed malformed, missing, or incorrect parameters to a tool. | |
| Missed Detail | Skipped a key part of the user prompt or prior context. | |
| Format & Instruction | Bad Format | Output is not valid JSON, CSV, or code. |
| Instruction Adherence | Didn’t follow instruction or style. |
Safety & Security Risks
Any output or behavior that may cause harm, leak personal data, or violate security best practices.| Subcategory | Error Type | Description |
|---|---|---|
| Ethical Violations | Unsafe Advice | Could lead to harm if followed. |
| PII Leak | Sensitive personal info exposed in output. | |
| Biased Output | Stereotyped, unfair, or discriminatory content. | |
| Security Failures | Token Exposure | Secrets, API keys, or auth tokens were exposed in output or logs. |
| Insecure API Usage | Used HTTP instead of HTTPS, skipped auth headers, or lacked rate limits. |
Tool & System Failures
Errors due to tool, API, environment, or runtime failures.| Subcategory | Error Type | Description |
|---|---|---|
| Setup Errors | Tool Missing | Tool not registered or available. |
| Tool Misconfigured | Tool or API setup is incorrect (e.g., bad schema, invalid registration). | |
| Env Incomplete | Missing tokens, secrets, or setup environment variables. | |
| Tool/API Failures | Rate Limit | Too many requests hit the limit. |
| Auth Fail | Authentication to tool or service failed. | |
| Server Crash | Tool/API returned internal error. | |
| Resource Not Found | Requested endpoint or resource does not exist or is not reachable. | |
| Runtime Limits | Out of Memory | RAM or resource limit breached. |
| Timeout | Execution took too long and was halted. |
Workflow & Task Gaps
Breakdowns in multi-step task execution, orchestration, or memory.| Subcategory | Error Type | Description |
|---|---|---|
| Context Loss | Dropped Context | Missed relevant past messages or data. |
| Overuse | Unnecessary context/tools invoked. | |
| Retrieval Errors | Poor Chunk Match | Retrieved irrelevant or unrelated context. |
| Wrong Chunk Used | Response based on wrong part of retrieved content. | |
| No Retrieval | Failed to run retrieval when needed. | |
| Task Flow Issues | Goal Drift | Strayed from intended objective. |
| Step Disorder | Steps executed out of logical order. | |
| Redundant Steps | Repeated same tool or action unnecessarily. | |
| Task Orchestration Failure | Agent failed to plan or interleave actions properly across tools or steps. | |
| Trace Completion | Incomplete Task | No final result or closure. |
Reflection Gaps
Agent failed to engage in introspective reasoning or revise steps appropriately.| Error Type | Description |
|---|---|
| Missing CoT | No intermediate thinking steps (Chain of Thought) were used to justify actions. |
| Missing ReAct Planning | Agent failed to interleave reasoning with action; took action without planning. |
| Lack of Self-Correction | Agent didn’t revise response or plan after detecting error or contradiction. |