Recall@K

Recall@K measures what fraction of all the relevant chunks for a query actually show up in the top K retrieved results. Run it to check whether your retriever is missing relevant context.

What it does

Recall@K is a statistical metric. It compares the retrieved chunks against the ground-truth relevant chunks and scores the fraction of relevant chunks that were found.

Input

Output

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

import json
from fi.evals import evaluate

result = evaluate(
    "recall_at_k",
    hypothesis=json.dumps([
        "Paris is the capital of France.",
        "The Eiffel Tower was built in 1889.",
        "France is in Europe.",
        "The Louvre is in Paris.",
        "Napoleon was born in Corsica."
    ]),
    reference=json.dumps([
        "Paris is the capital of France.",
        "The Eiffel Tower was built in 1889.",
        "The Louvre is in Paris."
    ]),
    eval_config={"k": 5},
)

print(result.score)   # 1.0
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "recall_at_k",
  {
    hypothesis: JSON.stringify([
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "France is in Europe.",
      "The Louvre is in Paris.",
      "Napoleon was born in Corsica."
    ]),
    reference: JSON.stringify([
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "The Louvre is in Paris."
    ])
  },
  { evalConfig: { k: 5 } }
);

console.log(result.score);   // 1.0
console.log(result.reason);

In this example, 5 chunks are retrieved and 3 are in the ground truth. With K set to 5 (the full list), all 3 relevant chunks appear in the retrieved results, giving a recall of 3/3 = 1.0. Try setting eval_config={"k": 3} to see how recall drops when only the top 3 chunks are considered.

Batch evaluation

To evaluate multiple queries in a single call, pass a list of JSON-serialized inputs. Each element represents one retrieval evaluation:

results = evaluate(
    "recall_at_k",
    hypothesis=[
        json.dumps(["Paris is the capital of France.", "France is in Europe.", "Napoleon was born in Corsica."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["Unrelated 1.", "Unrelated 2.", "Unrelated 3.", "The Louvre is in Paris."]),
    ],
    reference=[
        json.dumps(["Paris is the capital of France.", "The Eiffel Tower was built in 1889."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["The Louvre is in Paris."]),
    ],
    eval_config={"k": 3},
)

for i, r in enumerate(results):
    print(f"Query {i+1}: {r.score}")
# Query 1: 0.5   (1 of 2 relevant found in top 3)
# Query 2: 1.0   (2 of 2 relevant found)
# Query 3: 0.0   (relevant chunk at position 4, outside top 3)

How it works

Recall@K answers the question: of all the chunks that should have been retrieved, how many actually appear in the top K results?

Formula:

Recall@K = (number of relevant items in top K) / (total number of relevant items)

Matching is based on exact string equality between retrieved chunks and ground-truth chunks. A recall of 1.0 means the retriever found every relevant chunk; a recall of 0.5 means half of the relevant chunks are missing.

By default (without eval_config), the evaluator uses the full retrieved list. Pass eval_config={"k": N} to limit evaluation to the top N chunks.

When to use

Run Recall@K wherever missing relevant context is the bigger risk than returning some noise.

• RAG and retrieval pipelines, to check whether the retriever is surfacing all the relevant chunks for a query
• Tuning retriever parameters like top-K or chunk size, where you need a coverage metric to track against
• Comparing embedding models or retrieval strategies on the same ground-truth set

What to do when Recall@K fails

If recall is low, the retriever is missing relevant context:

• Increase the number of chunks retrieved (higher K) to capture more relevant results
• Improve the embedding model or chunking strategy so relevant content ranks higher
• Check if ground-truth chunks are being split across multiple smaller chunks, causing partial matches
• Ensure the query is being embedded with the same model used for document embeddings
• Consider hybrid retrieval (combining dense and sparse methods) to catch different types of relevance