Conversation Coherence

Evaluates how logically a conversation flows and whether responses stay consistent with earlier context.

Conversation Coherence checks whether a multi-turn exchange holds together: responses follow logically from what came before and don’t contradict or ignore earlier turns. Run it to catch dialogue that drifts or loses the thread.

What it does

Conversation Coherence is an LLM-as-Judge eval. It reads the full conversation and scores how logically it flows and how well context is maintained across turns.

Input

Required Input	Type	Description
`conversation`	`string`	Conversation history between the user and the model provided as query and response pairs

Output

Field	Type	Description
Result	`score`	Higher scores indicate more coherent conversation
Reason	`string`	A plain-language explanation of the conversation coherence assessment

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "conversation_coherence",
    conversation="User: My Wi-Fi keeps disconnecting every few minutes.\nAssistant: You can try restarting your router and updating your network drivers.\nUser: I restarted the router and it's stable now. Thanks!\nAssistant: Glad to hear that! Let me know if you need anything else.",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "conversation_coherence",
  {
    conversation: "User: My Wi-Fi keeps disconnecting every few minutes.\nAssistant: You can try restarting your router and updating your network drivers.\nUser: I restarted the router and it's stable now. Thanks!\nAssistant: Glad to hear that! Let me know if you need anything else."
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Conversation Coherence wherever a user interacts with your assistant across multiple turns.

Multi-turn chat and voice assistants, to confirm responses stay logically connected to earlier turns
Long support or sales conversations, where context needs to persist across the whole thread
Debugging dialogue that feels disjointed or drops earlier information

What to do when Conversation Coherence fails

Review the conversation history to identify where the context break occurred. Implement context window management so important information from earlier turns is retained through the rest of the conversation.

If context loss is persistent, consider reducing the length of conversation threads or adding explicit summarization between turns.

Was this page helpful?

Questions & Discussion