MRR

MRR (Mean Reciprocal Rank) measures how quickly the retriever surfaces the first relevant chunk. Run it when getting a relevant result to the top matters more than finding every relevant chunk.

What it does

MRR is a statistical metric. It compares the retrieved chunks against the ground-truth relevant chunks and scores the reciprocal of the position where the first relevant chunk appears.

Input

Output

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

import json
from fi.evals import evaluate

result = evaluate(
    "mrr",
    hypothesis=json.dumps([
        "France is in Europe.",
        "Napoleon was born in Corsica.",
        "Paris is the capital of France.",
        "The Eiffel Tower was built in 1889.",
        "The Louvre is in Paris."
    ]),
    reference=json.dumps([
        "Paris is the capital of France.",
        "The Eiffel Tower was built in 1889.",
        "The Louvre is in Paris."
    ]),
)

print(result.score)   # 0.333
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "mrr",
  {
    hypothesis: JSON.stringify([
      "France is in Europe.",
      "Napoleon was born in Corsica.",
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "The Louvre is in Paris."
    ]),
    reference: JSON.stringify([
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889.",
      "The Louvre is in Paris."
    ])
  }
);

console.log(result.score);   // 0.333
console.log(result.reason);

In this example, the first relevant chunk (“Paris is the capital of France.”) appears at position 3, so the reciprocal rank is 1/3 = 0.333.

Batch evaluation

To evaluate multiple queries in a single call, pass a list of JSON-serialized inputs. Each element represents one retrieval evaluation:

results = evaluate(
    "mrr",
    hypothesis=[
        json.dumps(["Paris is the capital of France.", "France is in Europe.", "Napoleon was born in Corsica."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["Unrelated 1.", "Unrelated 2.", "Unrelated 3.", "The Louvre is in Paris."]),
    ],
    reference=[
        json.dumps(["Paris is the capital of France.", "The Eiffel Tower was built in 1889."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["The Louvre is in Paris."]),
    ],
)

for i, r in enumerate(results):
    print(f"Query {i+1}: {r.score}")
# Query 1: 1.0    (first relevant at position 1)
# Query 2: 1.0    (first relevant at position 1)
# Query 3: 0.25   (first relevant at position 4)

How it works

MRR measures how quickly the retriever surfaces the first relevant result. The score is the reciprocal of the rank position where the first relevant chunk appears.

Formula:

MRR = 1 / (position of the first relevant item)

If the first relevant chunk is at position 1, the score is 1.0. At position 2, it’s 0.5. At position 3, it’s 0.333. If no relevant chunk is found, the score is 0.0.

MRR is particularly useful for question-answering RAG systems where the first relevant chunk often contains the answer. It directly measures the user experience of finding information quickly.

Matching is based on exact string equality between retrieved chunks and ground-truth chunks.

When to use

Run MRR wherever the user experience depends on finding a good answer fast, not on retrieving every relevant chunk.

• Question-answering RAG systems, where the first relevant chunk usually carries the answer
• Search and retrieval UX, to measure how quickly users would see a useful result
• Comparing retrieval strategies where speed-to-first-hit matters more than total coverage

What to do when MRR fails

If MRR is low, the first relevant chunk is appearing too far down in results:

• Apply a re-ranking step to push the most relevant chunk to the top position
• Check if irrelevant but semantically similar chunks are outranking the correct answer
• Ensure query formatting matches the style of your indexed documents
• For short queries, consider query expansion to add context that helps the retriever identify the best match
• Verify that the first relevant chunk in your ground truth is actually the most directly relevant one