Conversation Coherence

Evaluates how logically a conversation flows and maintains context throughout the dialogue. This metric assesses whether responses are consistent, contextually appropriate, and maintain a natural progression of ideas within the conversation thread. 

result = evaluator.evaluate(
    eval_templates="conversation_coherence",
    inputs={
        "conversation": '''
                    User: My Wi-Fi keeps disconnecting every few minutes.
                    Assistant: You can try restarting your router and updating your network drivers.
                    User: I restarted the router and it's stable now. Thanks!
                    Assistant: Glad to hear that! Let me know if you need anything else.
                  '''
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)

import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "conversation_coherence",
  {
    conversation: "User: My Wi-Fi keeps disconnecting every few minutes. Assistant: You can try restarting your router and updating your network drivers. User: I restarted the router and it's stable now. Thanks! Assistant: Glad to hear that! Let me know if you need anything else."
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
conversationstringConversation history between the user and the model provided as query and response pairs
Output
FieldDescription
ResultReturns a score, where higher scores indicate more coherent conversation
ReasonProvides a detailed explanation of the conversation coherence assessment

What to do when Conversation Coherence is Low

  • Review conversation history to identify where context breaks occurred
  • Implement context window management to ensure important information is retained
  • Consider reducing the length of conversation threads if context loss is persistent

Comparing Conversation Coherence with Similar Evals

  1. Conversation Resolution: While Coherence focuses on the flow and context maintenance throughout the conversation, Resolution evaluates whether the conversation reaches a satisfactory conclusion.
  2. Context Adherence: Coherence differs from Context Adherence as it evaluates the internal consistency of the conversation rather than adherence to external context.
  3. Completeness: Coherence focuses on the logical flow between messages, while Completeness evaluates whether individual responses fully address their queries.
Was this page helpful?

Questions & Discussion