Conversation Resolution
Checks whether a conversation reaches a satisfactory conclusion and the user's query actually gets resolved.
Conversation Resolution checks whether a conversation ends with the user’s need actually met, not just answered. Run it to catch dialogues that trail off, loop, or end before the user got what they came for.
What it does
Conversation Resolution is an LLM-as-Judge eval. It reads the full conversation and scores whether it reaches a satisfactory conclusion.
Input
| Required Input | Type | Description |
|---|---|---|
conversation | string | Conversation history between the user and the model provided as query and response pairs |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher scores indicate more resolved conversation |
| Reason | string | A plain-language explanation of the conversation resolution assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"conversation_resolution",
conversation="User: My Wi-Fi keeps disconnecting every few minutes.\nAssistant: You can try restarting your router and updating your network drivers.\nUser: I restarted the router and it's stable now. Thanks!\nAssistant: Glad to hear that! Let me know if you need anything else.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"conversation_resolution",
{
conversation: "User: My Wi-Fi keeps disconnecting every few minutes.\nAssistant: You can try restarting your router and updating your network drivers.\nUser: I restarted the router and it's stable now. Thanks!\nAssistant: Glad to hear that! Let me know if you need anything else."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Conversation Resolution wherever a conversation is expected to end with the user’s issue actually settled.
- Support and troubleshooting chats, to confirm the user’s problem was solved before the thread ends
- Voice and chat assistants, to catch conversations that end abruptly or leave the user without an answer
- Multi-turn flows where a satisfying conclusion matters as much as any single correct response
What to do when Conversation Resolution fails
Add confirmation mechanisms that verify user satisfaction before a conversation is treated as closed, and develop fallback responses for unclear or complex queries that would otherwise stall.
Track common patterns in unresolved queries to find recurring gaps, and consider adding a clarification system for ambiguous requests so they don’t dead-end.
Questions & Discussion