Evaluating RAG Applications

fi.evals ships in the ai-evaluation package, not futureagi.

from fi.evals import evaluate
from datasets import load_dataset

ragas-wikiqa pairs each question with retrieved context and an answer generated from that context, which is what a RAG eval needs.

dataset = load_dataset("explodinggradients/ragas-wikiqa")
sample_data = dataset["train"]

df = sample_data.to_pandas().head(10)
print(df[["question", "context", "generated_with_rag"]])

You should see 10 rows with question, context, and generated_with_rag columns populated.

Score the first row to see what a real result looks like before running the whole sample.

first = df.iloc[0]
result = evaluate(
    "context_relevance",
    input=first["question"],
    context=first["context"],
    model="turing_small",
)
print(result.score, result.reason)

You should see a score between 0 and 1 and a short text explanation.

Each metric needs a different combination of context, input, and output. Passing all three to every call, as Step 4 does, covers every metric here.

metrics = ["context_relevance", "completeness", "factual_accuracy"]

• context_relevance (needs context, input): are the retrieved chunks relevant to the question
• completeness (needs input, output): does the answer address every part of the question
• factual_accuracy (needs input, output, context): are the claims in the answer correct

Scale the single evaluate() call from Step 2 to every metric and every row.

for metric in metrics:
    df[metric] = None

for index, row in df.iterrows():
    for metric in metrics:
        result = evaluate(
            metric,
            input=row["question"],
            output=row["generated_with_rag"],
            context=row["context"],
            model="turing_small",
        )
        df.at[index, metric] = result.score

You should see each metric column filled with a score between 0 and 1 for all 10 rows.

for metric in metrics:
    print(f"Average {metric}: {df[metric].mean():.2f}")

Expected output (illustrative, your numbers depend on the sample and model version):

Average context_relevance: 0.81
Average completeness: 0.88
Average factual_accuracy: 0.76

factual_accuracy is the weakest of the three here, so that’s the metric worth digging into next.

An average hides which row is actually broken. Sort on the metric you care about and inspect the lowest scorer.

worst = df.sort_values("factual_accuracy").iloc[0]

print(f"Question: {worst['question']}")
print(f"Answer:   {worst['generated_with_rag']}")
print(f"Score:    {worst['factual_accuracy']}")

Read the answer against its context column. A low factual_accuracy score with a context that does contain the fact points at the generation step, not retrieval; a low score with missing context points at the retriever.

Change one thing and rerun: re-score the rows below the sample average with turing_large instead of turing_small, and compare the means.

before = df["factual_accuracy"].mean()

weak = df[df["factual_accuracy"] < before]
for index, row in weak.iterrows():
    result = evaluate(
        "factual_accuracy",
        input=row["question"],
        output=row["generated_with_rag"],
        context=row["context"],
        model="turing_large",
    )
    df.at[index, "factual_accuracy"] = result.score

after = df["factual_accuracy"].mean()
print(f"factual_accuracy: {before:.2f} -> {after:.2f}")

Expected output (illustrative, your numbers depend on the sample and model version):

factual_accuracy: 0.76 -> 0.84

turing_large catching claims turing_small missed is the usual driver of a delta like this; a rerun that doesn’t move the mean points back at retrieval instead.