Customer Agent: Query Handling
Assesses how effectively the agent interprets user queries and provides relevant, helpful answers.
Customer Agent Query Handling checks whether an agent correctly understands what the customer is asking and responds with a relevant, helpful answer. Run it to measure how well an agent handles the questions it’s actually built for.
What it does
Customer Agent Query Handling is an LLM-as-Judge eval. It reads the full conversation and scores how effectively the agent interprets and answers the customer’s queries.
Input
| Required Input | Type | Description |
|---|---|---|
conversation | string | The full conversation history between the customer and agent |
Output
| Field | Type | Description |
|---|---|---|
| Result | never / occasionally / frequently / always | Indicates how often the agent correctly handles queries |
| Reason | string | A plain-language explanation of the query handling assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"customer_agent_query_handling",
conversation="User: Can I return a product I bought last week?\nAgent: Yes, we have a 30-day return policy. You can initiate a return from your account page or visit any of our stores.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"customer_agent_query_handling",
{
conversation: "User: Can I return a product I bought last week?\nAgent: Yes, we have a 30-day return policy. You can initiate a return from your account page or visit any of our stores."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Customer Agent Query Handling to measure the core competence of a support agent: understanding and answering.
- Support bots fielding common product, billing, or policy questions
- Regression checks after a knowledge base or prompt update, to confirm answer quality didn’t drop
- Comparing agent versions on how well they interpret customer intent
What to do when Customer Agent Query Handling fails
Review cases where the agent misunderstood the user’s intent, and improve intent detection and query classification around those patterns. Expand the agent’s knowledge base with more relevant responses where gaps show up.
Add clarification prompts for ambiguous or complex queries instead of letting the agent guess.
Questions & Discussion