Customer Agent: Context Retention

Evaluates if the agent correctly retains and applies context from earlier in the conversation without asking for the same information again. 

result = evaluator.evaluate(
    eval_templates="customer_agent_context_retention",
    inputs={
        "conversation": "User: My name is Sarah and I have a question about my order #98765.\nAgent: Hi Sarah! I can help with order #98765. What's your question?\nUser: When will it arrive?\nAgent: Could you please provide your name and order number?"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "customer_agent_context_retention",
  {
    conversation: "User: My name is Sarah and I have a question about my order #98765.\nAgent: Hi Sarah! I can help with order #98765. What's your question?\nUser: When will it arrive?\nAgent: Could you please provide your name and order number?"
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
conversationstringThe full conversation history between the customer and agent
Output
FieldDescription
ResultReturns a numeric score from 0 to 100, where higher values indicate better context retention
ReasonProvides a detailed explanation of the context retention assessment

What to Do When Context Retention Score is Low

  • Ensure the agent’s memory window covers the full conversation length
  • Add explicit context summarization between turns
  • Review cases where the agent re-asks for information already provided
  • Implement entity tracking to persist key facts across the conversation

Comparing Context Retention with Similar Evals

  • Customer Agent: Loop Detection: Context Retention evaluates whether the agent remembers and applies earlier information, while Loop Detection identifies repetitive agent behavior.
  • Conversation Coherence: Context Retention focuses on memory across turns, while Conversation Coherence evaluates the overall logical flow of the dialogue.
Was this page helpful?

Questions & Discussion