Precision@K

Precision@K measures how much of what your retriever returns in the top K is actually relevant. Run it to quantify how much noise reaches the generator.

What it does

Precision@K is a statistical metric. It compares the retrieved chunks against the ground-truth relevant chunks and scores the fraction of the top K results that are relevant.

Input

Output

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

import json
from fi.evals import evaluate

result = evaluate(
    "precision_at_k",
    hypothesis=json.dumps([
        "Paris is the capital of France.",
        "France is in Europe.",
        "The Eiffel Tower was built in 1889.",
        "Napoleon was born in Corsica.",
        "The Louvre is in Paris."
    ]),
    reference=json.dumps([
        "Paris is the capital of France.",
        "The Eiffel Tower was built in 1889.",
        "The Louvre is in Paris."
    ]),
    eval_config={"k": 5},
)

print(result.score)   # 0.6
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "precision_at_k",
  {
    hypothesis: JSON.stringify([
      "Paris is the capital of France.",
      "France is in Europe.",
      "The Eiffel Tower was built in 1889.",
      "Napoleon was born in Corsica.",
      "The Louvre is in Paris."
    ]),
    reference: JSON.stringify([
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "The Louvre is in Paris."
    ])
  },
  { evalConfig: { k: 5 } }
);

console.log(result.score);   // 0.6
console.log(result.reason);

In this example, 5 chunks are retrieved. Of those 5, 3 are in the ground truth (“Paris is the capital…”, “The Eiffel Tower…”, and “The Louvre is in Paris.”), giving a precision of 3/5 = 0.6.

Batch evaluation

To evaluate multiple queries in a single call, pass a list of JSON-serialized inputs. Each element represents one retrieval evaluation:

results = evaluate(
    "precision_at_k",
    hypothesis=[
        json.dumps(["Paris is the capital of France.", "France is in Europe.", "Napoleon was born in Corsica."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["Unrelated 1.", "Unrelated 2.", "Unrelated 3.", "The Louvre is in Paris."]),
    ],
    reference=[
        json.dumps(["Paris is the capital of France.", "The Eiffel Tower was built in 1889."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["The Louvre is in Paris."]),
    ],
    eval_config={"k": 3},
)

for i, r in enumerate(results):
    print(f"Query {i+1}: {r.score}")
# Query 1: 0.333   (1 relevant in top 3 / 3)
# Query 2: 0.667   (2 relevant in top 3 / 3)
# Query 3: 0.0     (0 relevant in top 3 / 3)

How it works

Precision@K answers the question: of the top K chunks the retriever returned, how many are actually relevant?

Formula:

Precision@K = (number of relevant items in top K) / K

The denominator is always K, even if fewer than K items were retrieved. Matching is based on exact string equality between retrieved chunks and ground-truth chunks.

A precision of 1.0 means every retrieved chunk is useful; a precision of 0.5 means half the results are noise. Low precision means your LLM receives irrelevant context, which can increase cost (more tokens) and in some cases cause the model to hallucinate based on misleading information.

By default (without eval_config), the evaluator uses the full retrieved list. Pass eval_config={"k": N} to limit evaluation to the top N chunks.

When to use

Run Precision@K wherever noisy retrieved context is the concern, not just missing coverage.

• RAG and retrieval pipelines, to check how much of the retrieved context is actually useful to the generator
• Cost and token-budget tuning, since irrelevant chunks add tokens without adding value
• Comparing retriever configurations to see which one returns cleaner results

What to do when Precision@K fails

If precision is low, the retriever is returning too much irrelevant content:

• Reduce the number of chunks retrieved (lower K) to keep only the most confident matches
• Improve the embedding model to better distinguish relevant from irrelevant content
• Apply a similarity threshold to filter out low-confidence results before passing to the LLM
• Review your chunking strategy: chunks that are too large may contain a mix of relevant and irrelevant content
• Consider re-ranking retrieved results with a cross-encoder before passing them to the generator