Evaluator SDK Basics

import os
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

No output to check here. The Evaluator instance is what every later step calls .evaluate() on.

context_adherence is a judged template: it needs a model_name to act as the grading model.

support_context = (
    "Refunds are issued within 5-7 business days to the original payment method. "
    "Store credit is available immediately as an alternative."
)
support_reply = "You'll get your refund back on the original card within 5 to 7 business days."

result = evaluator.evaluate(
    eval_templates="context_adherence",
    inputs={"context": support_context, "output": support_reply},
    model_name="turing_flash",
)

item = result.eval_results[0]
print(item.output)
print(item.reason)

You should see something like this (illustrative, exact wording varies by run):

0.92
The response accurately reflects the refund timeline stated in the context.

item.output is the 0-1 adherence score, item.reason is the grading model’s explanation.

Deterministic templates don’t grade with a model. Pass eval_templates and inputs only, no model_name.

order_confirmation = '{"order_id": "SO-48213", "status": "shipped", "refund_issued": false}'

result = evaluator.evaluate(
    eval_templates="is_json",
    inputs={"text": order_confirmation},
)

print(result.eval_results[0].output)

You should see:

Passed

pii is a judged safety template. Run it on inbound text before it reaches a log or a downstream store.

customer_message = "My order number is SO-48213 and my name is Jordan Reyes."

result = evaluator.evaluate(
    eval_templates="pii",
    inputs={"input": customer_message},
    model_name="turing_flash",
)

item = result.eval_results[0]
print(item.output)
print(item.reason)

You should see the check fail the message and name what it found (illustrative):

Failed
The message contains a personal name (Jordan Reyes) and an order identifier.

Loop evaluate() over a list of candidate replies and compare scores directly.

support_replies = [
    "Your refund will be back on your card within 5 to 7 business days after we process the return.",
    "Sure, no problem at all, happy to help, let me just check on that for you real quick, one moment please.",
]

for reply in support_replies:
    result = evaluator.evaluate(
        eval_templates="is_concise",
        inputs={"output": reply},
        model_name="turing_flash",
    )
    item = result.eval_results[0]
    print(f"{item.output!s:<8} {reply[:50]}")

You should see the first reply pass and the second fail on filler (illustrative):

Passed   Your refund will be back on your card within 5 
Failed   Sure, no problem at all, happy to help, let me

The second reply fails on filler. Tighten it and rerun the same check:

tightened_reply = "Your refund will be back on your card within 5 to 7 business days."

result = evaluator.evaluate(
    eval_templates="is_concise",
    inputs={"output": tightened_reply},
    model_name="turing_flash",
)
item = result.eval_results[0]
print(f"before: Failed")
print(f"after:  {item.output!s}")

You should see the failure close:

before: Failed
after:  Passed

Wrap evaluate() so a bad template name or a transient API error doesn’t crash a batch job.

def score_reply(template, output, **extra_inputs):
    try:
        result = evaluator.evaluate(
            eval_templates=template,
            inputs={"output": output, **extra_inputs},
            model_name="turing_flash",
        )
        item = result.eval_results[0]
        return item.output, item.reason
    except Exception as exc:
        print(f"Eval '{template}' failed: {exc}")
        return None, None

status, reason = score_reply("toxicity", "I completely disagree, but I respect your view.")
print(status, reason)

You should see the toxicity check pass with no exception:

Passed The response is respectful and contains no toxic language.

The try/except here is what turns a bad template name into a logged failure instead of a stopped script.