Customer Agent: Context Retention
Evaluates if the agent correctly retains and applies context from earlier in the conversation without re-asking for it.
Customer Agent Context Retention checks whether an agent remembers what the user already told it, like their name or order number, instead of asking for it again. Run it to catch agents that lose track of earlier turns.
What it does
Customer Agent Context Retention is an LLM-as-Judge eval. It reads the full conversation and scores how well the agent retains and applies context from earlier turns.
Input
| Required Input | Type | Description |
|---|---|---|
conversation | string | The full conversation history between the customer and agent |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher values indicate better context retention |
| Reason | string | A plain-language explanation of the context retention assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"customer_agent_context_retention",
conversation="User: My name is Sarah and I have a question about my order #98765.\nAgent: Hi Sarah! I can help with order #98765. What's your question?\nUser: When will it arrive?\nAgent: Could you please provide your name and order number?",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"customer_agent_context_retention",
{
conversation: "User: My name is Sarah and I have a question about my order #98765.\nAgent: Hi Sarah! I can help with order #98765. What's your question?\nUser: When will it arrive?\nAgent: Could you please provide your name and order number?"
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Customer Agent Context Retention wherever a conversation carries information across turns that the agent needs to keep using.
- Support conversations that open with identifying details, like a name or order number, that later turns depend on
- Long conversations where account details, preferences, or earlier answers need to persist
- Debugging agents that appear to “forget” what the user just said
What to do when Customer Agent Context Retention fails
Ensure the agent’s memory window covers the full conversation length, and add explicit context summarization between turns so key facts aren’t dropped. Review cases where the agent re-asks for information already provided to find the pattern behind the loss.
Implement entity tracking to persist key facts, like names and order numbers, across the whole conversation.
Questions & Discussion