RAG Evaluation: Retrieval vs Generation

Define a realistic query, retrieved context chunks, and generated answer. This example simulates a company knowledge-base RAG system.

query = "What is the refund policy and how long does processing take?"

retrieved_context = (
    "Chunk 1: Customers may request a full refund within 30 days of purchase. "
    "Refunds are processed within 5-7 business days after approval. "
    "Chunk 2: To initiate a refund, contact support@example.com with your order number. "
    "Chunk 3: Gift cards and promotional items are non-refundable. "
    "Chunk 4: Our company was founded in 2015 and is headquartered in San Francisco."
)

generated_answer = (
    "You can request a full refund within 30 days of purchase. "
    "Once approved, refunds are processed in 5-7 business days. "
    "To start, email support@example.com with your order number. "
    "Gift cards and promotional items cannot be refunded."
)

Chunk 4 is irrelevant to the query, a common retrieval problem. The metrics below surface it.

You should see: nothing yet, this step only defines the fixtures the rest of the tutorial scores.

evaluate() takes an eval name (or a list of eval names), the fields that eval needs as keyword arguments, and a model (or engine) to run it on. Each of the five RAG metrics reads a different combination of context, input, and output:

For hallucination-specific metrics (faithfulness, groundedness, and context_adherence), see Hallucination Detection.

Run the three retrieval metrics individually first, so each score and reason is easy to read on its own.

from fi.evals import evaluate

# Context relevance: are the retrieved chunks relevant to the query?
relevance = evaluate(
    "context_relevance",
    context=retrieved_context,
    input=query,
    model="turing_flash",
)
print(f"Context Relevance: score={relevance.score}, passed={relevance.passed}")
print(f"  Reason: {relevance.reason}\n")

# Chunk attribution: was the context chunk used in the response?
attribution = evaluate(
    "chunk_attribution",
    context=retrieved_context,
    output=generated_answer,
    model="turing_flash",
)
print(f"Chunk Attribution: score={attribution.score}, passed={attribution.passed}")
print(f"  Reason: {attribution.reason}\n")

# Chunk utilization: how effectively does the response use the context?
utilization = evaluate(
    "chunk_utilization",
    context=retrieved_context,
    output=generated_answer,
    model="turing_flash",
)
print(f"Chunk Utilization: score={utilization.score}, passed={utilization.passed}")
print(f"  Reason: {utilization.reason}\n")

You should see (illustrative, scores and reasons vary by run):

Context Relevance: score=0.75, passed=True
  Reason: Three of four chunks are relevant to the query; Chunk 4 is unrelated.

Chunk Attribution: score=1.0, passed=True
  Reason: Every claim in the output maps to a specific context chunk.

Chunk Utilization: score=0.75, passed=True
  Reason: The output uses content from 3 of 4 retrieved chunks.

evaluate() called with a single eval name returns an EvalResult, with .score, .passed, .reason, .eval_name, .latency_ms, .status, and .error fields.

These metrics evaluate whether the LLM fully addressed the query and produced factually accurate claims. To make the diagnostic concrete, score a deliberately wrong answer first, then fix it and rerun.

from fi.evals import evaluate

# Completeness: does the response fully address the query?
completeness = evaluate(
    "completeness",
    input=query,
    output=generated_answer,
    model="turing_flash",
)
print(f"Completeness: score={completeness.score}, passed={completeness.passed}")
print(f"  Reason: {completeness.reason}\n")

# Factual accuracy: introduce a claim the context doesn't support.
bad_answer = (
    "You can request a full refund within 30 days of purchase. "
    "Refunds are processed within 24 hours after approval. "
    "To start, email support@example.com with your order number."
)
bad_accuracy = evaluate(
    "factual_accuracy",
    input=query,
    output=bad_answer,
    context=retrieved_context,
    model="turing_flash",
)
print(f"Factual Accuracy (bad answer): score={bad_accuracy.score}, passed={bad_accuracy.passed}")
print(f"  Reason: {bad_accuracy.reason}\n")

# Fix: match the processing time actually stated in the context, then rerun.
accuracy = evaluate(
    "factual_accuracy",
    input=query,
    output=generated_answer,
    context=retrieved_context,
    model="turing_flash",
)
print(f"Factual Accuracy (fixed): score={accuracy.score}, passed={accuracy.passed}")
print(f"  Reason: {accuracy.reason}\n")

You should see (illustrative):

Completeness: score=1.0, passed=True
  Reason: The response fully addresses the query including refund eligibility, processing time, and exceptions.

Factual Accuracy (bad answer): score=0.25, passed=False
  Reason: The context states refunds are processed within 5-7 business days, not 24 hours; this claim is unsupported.

Factual Accuracy (fixed): score=1.0, passed=True
  Reason: All stated facts are accurate and confirmed by the provided context.

Fixing the one unsupported claim moves factual_accuracy from score=0.25, passed=False to score=1.0, passed=True.

Pass a list of eval names to score several metrics in one call. The result comes back as an iterable BatchResult.

from fi.evals import evaluate

diagnostic = evaluate(
    [
        "context_relevance",
        "chunk_attribution",
        "chunk_utilization",
        "completeness",
        "factual_accuracy",
    ],
    input=query,
    output=generated_answer,
    context=retrieved_context,
    model="turing_flash",
)

print("=== RAG Pipeline Diagnostic ===\n")
for result in diagnostic:
    print(f"{result.eval_name:<22} score={result.score} passed={result.passed}")
    print(f"  Reason: {result.reason}\n")

You should see (illustrative):

=== RAG Pipeline Diagnostic ===

context_relevance      score=0.75 passed=True
  Reason: Three of four chunks are relevant; Chunk 4 is off-topic.

chunk_attribution      score=1.0 passed=True
  Reason: Every output claim maps to a specific context chunk.

chunk_utilization      score=0.75 passed=True
  Reason: Output uses 3 of 4 chunks; Chunk 4 is unused.

completeness           score=1.0 passed=True
  Reason: The response fully addresses all parts of the query.

factual_accuracy       score=1.0 passed=True
  Reason: All stated facts are accurate and confirmed by the context.

Use the diagnostic output to decide where to focus your effort.

# Build a name-keyed lookup, then gate on the real passed field.
scores = {result.eval_name: result for result in diagnostic}

retrieval_ok = scores["context_relevance"].passed and scores["chunk_utilization"].passed
generation_ok = scores["completeness"].passed and scores["factual_accuracy"].passed

if not retrieval_ok:
    print("Action: improve retrieval. Check embeddings, re-ranking, or top-k settings.")
elif not generation_ok:
    print("Action: improve generation. Tune the prompt, lower temperature, or switch models.")
else:
    print("Pipeline healthy.")

You should see: Pipeline healthy. for the example data above, since all five metrics pass.