Eval Correction Loop

Start with a built-in template like is_helpful. It gives you a baseline plus the explanations the evaluator model used. The explanations are what you inspect in step 2.

import os
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

# Real support replies from your agent. The bad ones look helpful but
# violate domain-specific rules (refund escalation, no upsells) that
# a generic helpfulness eval has no way to know about.
samples = [
    {
        "id": "r1",
        "user_query": "I forgot my password and can't log in.",
        "agent_response": "Click 'Forgot password' on the sign-in page and follow the email link. If the email doesn't arrive within 5 minutes, reply here with a screenshot and I'll resend it from our end.",
        "human_verdict": "good",
    },
    {
        "id": "r2",
        "user_query": "I want a refund for last month's subscription. The service was down 3 times.",
        "agent_response": "I'm sorry to hear about the disruption. I'll process a full refund for last month's $49 today, and you'll see it in your account within 7 to 10 business days.",
        "human_verdict": "bad",  # all refund offers must be escalated to a manager per policy
    },
    {
        "id": "r3",
        "user_query": "Why is my bill higher than expected this month?",
        "agent_response": "Your Pro plan has overage fees for usage above 10GB. You used 15GB this billing cycle, which added $12 to your bill. By the way, our Premium plan includes unlimited storage and the new collaboration features. Want me to upgrade you?",
        "human_verdict": "bad",  # answer is correct but the upsell is a brand-voice violation
    },
    {
        "id": "r4",
        "user_query": "How do I export my data?",
        "agent_response": "Go to Settings > Data > Export. You can pick CSV or JSON, and we'll email a download link within 30 minutes.",
        "human_verdict": "good",
    },
]

baseline_results = []
for s in samples:
    r = evaluator.evaluate(
        eval_templates="is_helpful",
        inputs={"input": s["user_query"], "output": s["agent_response"]},
        model_name="turing_flash",
    )
    baseline_results.append({
        "id": s["id"],
        "eval_score": r.eval_results[0].output,
        "eval_reason": r.eval_results[0].reason,
        "human_verdict": s["human_verdict"],
    })

for row in baseline_results:
    print(f"{row['id']}: eval={row['eval_score']!s:>5} | human={row['human_verdict']:>4} | {row['eval_reason'][:80]}")

You should see is_helpful return Passed for r2 and r3. Both replies are on-topic, well-formed, and offer a concrete action, so nothing about the surface form gives the generic evaluator model a reason to fail them. Your team flags them as bad because they violate domain rules the evaluator model has no way to know about. That’s the disagreement signal the correction loop fixes.

A disagreement is any row where the eval and the human reach different verdicts. These are the rows that teach the evaluator something new.

def passed(score):
    return str(score).strip().lower() == "passed"

disagreements = [
    r for r in baseline_results
    if passed(r["eval_score"]) != (r["human_verdict"] == "good")
]

print(f"{len(disagreements)} / {len(baseline_results)} disagreed with humans")
for r in disagreements:
    print(f"  {r['id']}: eval said {r['eval_score']}, human said {r['human_verdict']}")
    print(f"    reason: {r['eval_reason'][:120]}")

You should see 2 disagreements: r2 and r3. Pick the rows that capture distinct failure modes (here: an off-policy refund promise and an in-support upsell). Those become your few-shot examples in the next step.

Create a custom eval template whose instructions spell out your domain’s definition of “good” and include the corrected examples inline. The evaluator model uses the examples to calibrate its decisions on new rows.

from fi.evals import EvalTemplateManager

rule_prompt = """\
You evaluate customer-support replies for a SaaS product.

A reply passes ONLY if ALL of the following hold:
1. Stays focused on the user's specific issue. No marketing language, no upsells, no pivots to other products.
2. Gives a concrete next step (a procedure, a link, a timeline, or a specific owner).
3. Does NOT commit to a refund, credit, or policy exception. Front-line agents must acknowledge the request and escalate to a manager.
4. Does NOT instruct the user to disable security features (2FA, MFA, encryption) as a workaround.

Examples of FAIL replies (learn from these):
- "I'm sorry to hear about the disruption. I'll process a full refund for last month's $49 today, and you'll see it in your account within 7 to 10 business days."
  -> FAIL: rule 3. Front-line agents can't commit to refunds. Should acknowledge and escalate.
- "Your Pro plan has overage fees for usage above 10GB. You used 15GB this billing cycle, which added $12 to your bill. By the way, our Premium plan includes unlimited storage and the new collaboration features. Want me to upgrade you?"
  -> FAIL: rule 1. Pivots from billing question to a sales pitch.

Example of a PASS reply:
- "Click 'Forgot password' on the sign-in page and follow the email link. If the email doesn't arrive within 5 minutes, reply here with a screenshot and I'll resend it from our end."
  -> PASS: focused on the issue, concrete next step, clear escalation path.

Now evaluate this reply.

User query: {{user_query}}
Agent response: {{agent_response}}
"""

template_manager = EvalTemplateManager(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

response = template_manager.create_template(
    name="support_reply_quality_v1",
    instructions=rule_prompt,
    model="turing_flash",
    output_type="pass_fail",
)
print(response)
# TemplateCreateResponse(id="<uuid>", name="support_reply_quality_v1", version="1")
# `name` is what you pass to `evaluator.evaluate(eval_templates=...)` in the next step.
# The required keys (user_query, agent_response) are inferred from the {{...}}
# placeholders in `instructions`, you don't declare them separately.

The examples above are inlined directly in instructions on purpose, so the full prompt is visible in one place. create_template also accepts a few_shot_examples argument if you’d rather pass them as structured data instead.

Two things make this work:

• The instructions enumerate the domain rules explicitly, so the evaluator model has criteria instead of vibes
• The few-shot examples cover the exact failure modes you found in step 2, so the evaluator model sees what “FAIL” looks like for your domain

Run the new eval on the same samples and compare against your human verdicts.

calibrated_results = []
for s in samples:
    r = evaluator.evaluate(
        eval_templates="support_reply_quality_v1",
        inputs={"user_query": s["user_query"], "agent_response": s["agent_response"]},
    )
    calibrated_results.append({
        "id": s["id"],
        "eval_score": r.eval_results[0].output,
        "human_verdict": s["human_verdict"],
    })

agreement = sum(
    1 for r in calibrated_results
    if passed(r["eval_score"]) == (r["human_verdict"] == "good")
)
print(f"agreement: {agreement} / {len(samples)} ({100 * agreement / len(samples):.0f}%)")
for r in calibrated_results:
    match = "OK" if passed(r["eval_score"]) == (r["human_verdict"] == "good") else "MISS"
    print(f"  {match} {r['id']}: eval={r['eval_score']} human={r['human_verdict']}")

Expect agreement to jump from around 50% baseline toward 100% on this set (illustrative for this 4-row batch, your own numbers depend on the batch and model). r2 and r3 now fail correctly because the instructions explicitly forbid out-of-policy refund commits and in-support upsells. is_helpful had no way to know either rule existed.

If agreement is still below where you need it (pick your own bar; teams commonly land around 85% on a held-out batch), the loop continues.

1. Pull a fresh sample of 20 to 30 rows the eval hasn’t seen
2. Re-score with the latest version (support_reply_quality_v1)
3. Find the new disagreements. These are failure modes your instructions didn’t cover
4. Rev to _v2: add 1 or 2 new few-shot examples or sharpen one of the rules. Avoid bloating. Every example added trades calibration for prompt length and inference cost.

# After collecting fresh disagreements...
rule_prompt_v2 = rule_prompt + """

Additional FAIL example (learn from this):
- "Try disabling 2FA temporarily so you can log in, then re-enable it once you're past the issue."
  -> FAIL: rule 4. Never instruct users to disable security features. Offer a recovery code or escalate to security ops.
"""

# Re-register with template_manager.create_template(name="support_reply_quality_v2", ...)
# and compare scores side-by-side.

Most evals stop moving after two or three iterations on a batch this size. Stop when fresh batches stay above your agreement bar. Adding more examples beyond that hurts more than it helps.