Customer Agent: Conversation Quality
Conversation-level quality metric assessing overall user experience: clarity, helpfulness, responsiveness, and tone.
Customer Agent Conversation Quality rates the overall experience of a support interaction rather than any single response. Run it to get a holistic read on how a conversation felt to the customer.
What it does
Customer Agent Conversation Quality is an LLM-as-Judge eval. It reads the full conversation and scores its overall quality across clarity, helpfulness, responsiveness, and tone.
Input
| Required Input | Type | Description |
|---|---|---|
conversation | string | The full conversation history between the customer and agent |
Output
| Field | Type | Description |
|---|---|---|
| Result | 1 / 2 / 3 / 4 / 5 | 1 is very poor and 5 is excellent overall conversation quality |
| Reason | string | A plain-language explanation of the conversation quality assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"customer_agent_conversation_quality",
conversation="User: Hi, I need help resetting my password.\nAgent: Of course! I'll send a reset link to your registered email. Is there anything else I can help you with?\nUser: That's all, thanks!\nAgent: You're welcome! Have a great day.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"customer_agent_conversation_quality",
{
conversation: "User: Hi, I need help resetting my password.\nAgent: Of course! I'll send a reset link to your registered email. Is there anything else I can help you with?\nUser: That's all, thanks!\nAgent: You're welcome! Have a great day."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Customer Agent Conversation Quality as a top-line score for customer-facing agent conversations.
- Dashboards and QA sampling, to track overall conversation quality across a support queue
- Comparing agent versions or prompt changes on the full interaction, not just one response
- Flagging low-scoring conversations for deeper review with the more specific customer agent evals
What to do when Customer Agent Conversation Quality fails
Review the full conversation for clarity, tone, and helpfulness, and identify the specific turns where the agent failed to meet user expectations. Improve response templates for the common customer scenarios that keep scoring low.
Combine this eval with the other customer agent evals, like Query Handling or Context Retention, to pinpoint the specific weakness behind a low score.
Questions & Discussion