Hallucination Detection

faithfulness checks whether a response is consistent with its retrieved context, no contradictions allowed. It runs entirely on a local NLI model, no API key required.

from fi.evals import evaluate

context = (
    "Pro plan subscriptions can be canceled anytime from Account Settings > Billing. "
    "Cancellation takes effect at the end of the current billing cycle, and no partial "
    "refunds are issued for unused time. Annual plans can be downgraded to monthly "
    "only at renewal, not mid-cycle."
)

question = "How do I cancel my Pro plan, and will I get a refund for the unused time?"

# A response that faithfully reflects the context
response = (
    "You can cancel your Pro plan anytime from Account Settings > Billing. "
    "Cancellation takes effect at the end of your current billing cycle, and you won't "
    "get a refund for the unused time."
)

result = evaluate(
    "faithfulness",
    output=response,
    context=context,
    input=question,
)

print(f"Faithfulness score : {result.score:.2f}")
print(f"Passed             : {result.passed}")
print(f"Reason             : {result.reason}")

Expected output (illustrative: exact wording depends on the installed NLI model):

Faithfulness score : 1.00
Passed             : True
Reason             : 2/2 claims supported

Every claim in the response traces back to the context, so the score is a perfect 1.00.

Run the same check against a response that contradicts the context on both cancellation timing and the refund.

from fi.evals import evaluate

hallucinated_response = (
    "You can cancel your Pro plan anytime, and it takes effect immediately. "
    "You'll receive a prorated refund for the unused portion of the billing cycle."
)

result = evaluate(
    "faithfulness",
    output=hallucinated_response,
    context=context,
    input=question,
)

print(f"Faithfulness score : {result.score:.2f}")
print(f"Passed             : {result.passed}")
print(f"Reason             : {result.reason}")

Expected output (illustrative: exact wording depends on the installed NLI model):

Faithfulness score : 0.00
Passed             : False
Reason             : 0/2 claims supported

Neither claim matches the context, so faithfulness fails the response and reason names both mismatches.

groundedness catches something faithfulness doesn’t: plausible-sounding additions that have no basis in the context at all, rather than direct contradictions. Run it through a Turing model.

from fi.evals import evaluate

# A response that adds a fact not present in the context
ungrounded_response = (
    "You can cancel your Pro plan anytime from Account Settings > Billing. "
    "Cancellation takes effect at the end of your current billing cycle, with no refund "
    "for unused time. Canceling also removes you from the referral rewards program."
)

result = evaluate(
    "groundedness",
    output=ungrounded_response,
    context=context,
    input=question,
    model="turing_small",
)

print(f"Passed : {result.passed}")
print(f"Reason : {result.reason}")

Expected output (illustrative: Turing’s judgment is model-based, so exact wording varies):

Passed : False
Reason : The response includes a claim that is not supported by the provided context.

The referral-program claim isn’t in the context, so groundedness fails the response even though the other two sentences are accurate.

Run the same check on a response that stays entirely within the context, to confirm the metric passes clean output.

from fi.evals import evaluate

clean_response = (
    "You can cancel your Pro plan anytime from Account Settings > Billing. "
    "Cancellation takes effect at the end of your current billing cycle, and there's no "
    "refund for unused time."
)

result = evaluate(
    "groundedness",
    output=clean_response,
    context=context,
    input=question,
    model="turing_small",
)

print(f"Passed : {result.passed}")
print(f"Reason : {result.reason}")

Expected output (illustrative: Turing’s judgment is model-based, so exact wording varies):

Passed : True
Reason : All claims are traceable to the provided context.

Pass a list of metric names to run faithfulness and groundedness together on the same output. evaluate() returns a BatchResult you can iterate or index by name.

from fi.evals import evaluate

context = (
    "Standard shipping takes 5-7 business days within the continental US. Expedited "
    "shipping is available for an additional $12.99 and delivers in 2-3 business days. "
    "International shipping is not currently supported. Orders ship Monday through "
    "Friday, excluding federal holidays."
)

question = "What shipping options are available and how long does each take?"

response = (
    "Standard shipping takes 5-7 business days within the continental US. For an "
    "additional $12.99, expedited shipping delivers in 2-3 business days. Orders ship "
    "Monday through Friday, excluding federal holidays."
)

results = evaluate(
    ["faithfulness", "groundedness"],
    output=response,
    context=context,
    input=question,
)

# Iterate over both results
for result in results:
    status = "PASS" if result.passed else "FAIL"
    if result.eval_name == "groundedness":
        # groundedness is a pass/fail check here; see steps 3-4
        print(f"{result.eval_name:<15} {status}")
    else:
        print(f"{result.eval_name:<15} score={result.score:.2f}  {status}")
    print(f"  Reason: {result.reason}")
    print()

# Or look up by name directly
faith_result = results.get("faithfulness")
ground_result = results.get("groundedness")

print(f"Both metrics passed: {faith_result.passed and ground_result.passed}")

Expected output (illustrative):

faithfulness    score=1.00  PASS
  Reason: 3/3 claims supported

groundedness    PASS
  Reason: All claims traceable to context

Both metrics passed: True

Both metrics pass because every claim in the response is both consistent with and traceable to the context.