Customer Agent: Query Handling

Assesses how effectively the agent correctly interprets user queries and gives relevant, helpful answers. 

result = evaluator.evaluate(
    eval_templates="customer_agent_query_handling",
    inputs={
        "conversation": "User: Can I return a product I bought last week?\nAgent: Yes, we have a 30-day return policy. You can initiate a return from your account page or visit any of our stores."
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "customer_agent_query_handling",
  {
    conversation: "User: Can I return a product I bought last week?\nAgent: Yes, we have a 30-day return policy. You can initiate a return from your account page or visit any of our stores."
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
conversationstringThe full conversation history between the customer and agent
Output
FieldDescription
ResultReturns one of: never, occasionally, frequently, or always — indicating how often the agent correctly handles queries
ReasonProvides a detailed explanation of the query handling assessment

What to Do When Query Handling is Poor

  • Review cases where the agent misunderstood the user’s intent
  • Improve intent detection and query classification
  • Expand the agent’s knowledge base with more relevant responses
  • Add clarification prompts for ambiguous or complex queries

Comparing Query Handling with Similar Evals

  • Customer Agent: Clarification Seeking: Query Handling evaluates whether the agent answers correctly, while Clarification Seeking checks if the agent appropriately asks for more information when needed.
  • Is Helpful: Query Handling is specific to customer agent conversations, while Is Helpful evaluates general response helpfulness for any input-output pair.
Was this page helpful?

Questions & Discussion