Multi-Turn Conversation Eval

The Evaluator client reads both keys and is reused for every metric below.

import os
import json
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

The constructor makes no network call, so a clean import and no traceback is the success signal here; a bad key surfaces on the first evaluate() call in Step 3.

A well-handled support conversation and a poorly-handled one. Both start with the same customer issue: a duplicate charge.

good_conversation = [
    {"role": "user", "content": "Hi, I was charged twice for order #7291. Can you help?"},
    {"role": "assistant", "content": "I'm sorry about that! Let me look up order #7291 right away."},
    {"role": "assistant", "content": "I can see the duplicate charge. I've initiated a refund of $49.99 to your original payment method. It should appear within 3-5 business days."},
    {"role": "user", "content": "Great, thank you. Will I get a confirmation email?"},
    {"role": "assistant", "content": "Yes, you'll receive a refund confirmation email within the next hour. Is there anything else I can help with?"},
    {"role": "user", "content": "No, that's all. Thanks for the quick help!"},
    {"role": "assistant", "content": "You're welcome! Don't hesitate to reach out if you need anything else. Have a great day!"},
]

bad_conversation = [
    {"role": "user", "content": "Hi, I was charged twice for order #7291. Can you help?"},
    {"role": "assistant", "content": "I'd be happy to help! Have you tried checking your account settings?"},
    {"role": "user", "content": "This isn't about account settings. I was charged $49.99 twice for the same order."},
    {"role": "assistant", "content": "I understand. Could you provide your order number?"},
    {"role": "user", "content": "I already said it's order #7291."},
    {"role": "assistant", "content": "Could you provide your order number so I can look into this?"},
    {"role": "user", "content": "I just told you twice, #7291. Can I speak to a manager?"},
    {"role": "assistant", "content": "I apologize for the inconvenience. Have you considered upgrading to our Premium support plan for faster resolution?"},
    {"role": "user", "content": "No! I just want my refund. Please transfer me to someone who can help."},
    {"role": "assistant", "content": "I understand your concern. Let me check your account settings to see if there's an issue."},
]

The bad conversation has multiple failure modes. It ignores the actual question, forgets the order number and asks twice, loops back to “account settings”, tries to upsell, and doesn’t escalate when the customer asks for a manager.

customer_agent_conversation_quality rates the overall interaction on a 1-5 scale considering clarity, helpfulness, responsiveness, and tone.

for label, convo in [("Good", good_conversation), ("Bad", bad_conversation)]:
    result = evaluator.evaluate(
        eval_templates="customer_agent_conversation_quality",
        inputs={"conversation": json.dumps(convo)},
        model_name="turing_small",
    )
    eval_result = result.eval_results[0]
    score = eval_result.output[0] if isinstance(eval_result.output, list) else eval_result.output
    print(f"{label} conversation: {score}/5")
    print(f"  Reason: {eval_result.reason}\n")

You should see output shaped like this (illustrative scores, your model call will vary):

Good conversation: 5/5
  Reason: The agent promptly addressed the issue, provided a clear resolution...

Bad conversation: 1/5
  Reason: The agent repeatedly ignored the customer's request, forgot context...

Run targeted metrics on the bad conversation to pinpoint specific failure modes.

Context retention: did the agent remember details from earlier in the conversation?

result = evaluator.evaluate(
    eval_templates="customer_agent_context_retention",
    inputs={"conversation": json.dumps(bad_conversation)},
    model_name="turing_small",
)
eval_result = result.eval_results[0]
print(f"Context retention: {eval_result.output}")
print(f"Reason: {eval_result.reason}")

Query handling: did the agent correctly interpret and answer the customer’s questions?

result = evaluator.evaluate(
    eval_templates="customer_agent_query_handling",
    inputs={"conversation": json.dumps(bad_conversation)},
    model_name="turing_small",
)
eval_result = result.eval_results[0]
score = eval_result.output[0] if isinstance(eval_result.output, list) else eval_result.output
print(f"Query handling: {score}")
print(f"Reason: {eval_result.reason}")

Loop detection: did the agent get stuck repeating the same prompts?

result = evaluator.evaluate(
    eval_templates="customer_agent_loop_detection",
    inputs={"conversation": json.dumps(bad_conversation)},
    model_name="turing_small",
)
eval_result = result.eval_results[0]
score = eval_result.output[0] if isinstance(eval_result.output, list) else eval_result.output
print(f"Loop detection: {score}")
print(f"Reason: {eval_result.reason}")

Human escalation: did the agent escalate when the customer asked for a manager?

result = evaluator.evaluate(
    eval_templates="customer_agent_human_escalation",
    inputs={"conversation": json.dumps(bad_conversation)},
    model_name="turing_small",
)
eval_result = result.eval_results[0]
score = eval_result.output[0] if isinstance(eval_result.output, list) else eval_result.output
print(f"Human escalation: {score}")
print(f"Reason: {eval_result.reason}")

You should see four verdicts, each naming a different failure (illustrative):

Context retention: 12
Reason: The agent asked for the order number twice despite the user providing it...

Query handling: never
Reason: The agent never directly addressed the duplicate charge issue...

Loop detection: frequently
Reason: The agent circled back to "account settings" twice and asked for the order number twice...

Human escalation: Failed
Reason: The user explicitly requested a manager but the agent deflected with an upsell...

Each metric catches a different dimension of failure. Together they tell a clear story: the agent forgot context, ignored the question, looped, and refused to escalate.

customer_agent_prompt_conformance checks whether the agent followed its system prompt throughout the conversation. It’s the only conversation metric that takes an additional system_prompt input.

system_prompt = (
    "You are a billing support agent for TechStore. "
    "Your role is to help customers resolve payment and billing issues. "
    "Always acknowledge the customer's issue first, then investigate. "
    "Never upsell products during a support interaction. "
    "If a customer asks to speak with a manager, escalate immediately."
)

for label, convo in [("Good", good_conversation), ("Bad", bad_conversation)]:
    result = evaluator.evaluate(
        eval_templates="customer_agent_prompt_conformance",
        inputs={
            "system_prompt": system_prompt,
            "conversation": json.dumps(convo),
        },
        model_name="turing_small",
    )
    eval_result = result.eval_results[0]
    score = eval_result.output[0] if isinstance(eval_result.output, list) else eval_result.output
    print(f"{label} conversation - prompt conformance: {score}")
    print(f"  Reason: {eval_result.reason}\n")

You should see the good conversation score high and the bad one score low (illustrative):

Good conversation - prompt conformance: 95
  Reason: The agent acknowledged the issue, investigated, and resolved it...

Bad conversation - prompt conformance: 8
  Reason: The agent violated multiple system prompt rules: upsold a product, failed to escalate...

Run 7 key metrics on both conversations in a single diagnostic sweep.

metrics = [
    ("conversation_coherence", "Coherence"),
    ("conversation_resolution", "Resolution"),
    ("customer_agent_conversation_quality", "Quality"),
    ("customer_agent_context_retention", "Context"),
    ("customer_agent_query_handling", "Queries"),
    ("customer_agent_loop_detection", "Loops"),
    ("customer_agent_human_escalation", "Escalation"),
]

print(f"{'Metric':<14}  {'Good':>12}  {'Bad':>12}")
print("-" * 42)

for metric_name, label in metrics:
    good_result = evaluator.evaluate(
        eval_templates=metric_name,
        inputs={"conversation": json.dumps(good_conversation)},
        model_name="turing_small",
    )
    bad_result = evaluator.evaluate(
        eval_templates=metric_name,
        inputs={"conversation": json.dumps(bad_conversation)},
        model_name="turing_small",
    )
    good_raw = good_result.eval_results[0].output
    bad_raw = bad_result.eval_results[0].output
    good_val = good_raw[0] if isinstance(good_raw, list) else good_raw
    bad_val = bad_raw[0] if isinstance(bad_raw, list) else bad_raw
    print(f"{label:<14}  {str(good_val):>12}  {str(bad_val):>12}")

You should see the good conversation pass every metric and the bad one fail across the board (illustrative scores):

Metric            Good           Bad
------------------------------------------
Coherence          1.0           0.4
Resolution         1.0           0.0
Quality            5/5           1/5
Context             95             12
Queries         always         never
Loops            never     frequently
Escalation      Passed        Failed

You can run all 10 conversational agent metrics at once from the dashboard using the Conversational agent evaluation eval group, no code required.

1. Go to app.futureagi.com and open Dataset
2. Open a dataset that has a conversation column (a JSON array of role/content messages) and a system_prompt column with the agent’s system prompt
3. Click Evaluate then Add Evaluations
4. Under Groups, select Conversational agent evaluation. This adds all 10 metrics in one click
5. Map the conversation column to the conversation input, and the system_prompt column to the system prompt input. This is needed for customer_agent_prompt_conformance, which checks whether the agent followed its instructions
6. Click Add & Run

All metrics run in parallel. Scores appear as new columns alongside your data, one column per metric. Most metrics only need the conversation column; the system_prompt mapping is used by customer_agent_prompt_conformance and ignored by the rest.