Hit Rate

Hit Rate answers the simplest retrieval question: did the retriever find at least one relevant chunk at all? Run it as a baseline sanity check before looking at finer-grained metrics.

What it does

Hit Rate is a statistical metric. It compares the retrieved chunks against the ground-truth relevant chunks and returns whether any match was found.

Input

Output

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

import json
from fi.evals import evaluate

result = evaluate(
    "hit_rate",
    hypothesis=json.dumps([
        "France is in Europe.",
        "Paris is the capital of France.",
        "Napoleon was born in Corsica."
    ]),
    reference=json.dumps([
        "Paris is the capital of France.",
        "The Eiffel Tower was built in 1889."
    ]),
)

print(result.score)   # 1.0
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "hit_rate",
  {
    hypothesis: JSON.stringify([
      "France is in Europe.",
      "Paris is the capital of France.",
      "Napoleon was born in Corsica."
    ]),
    reference: JSON.stringify([
      "Paris is the capital of France.",
      "The Eiffel Tower was built in 1889."
    ])
  }
);

console.log(result.score);   // 1.0
console.log(result.reason);

In this example, “Paris is the capital of France.” appears in both the retrieved and ground-truth lists, so at least one relevant chunk was found: hit rate = 1.0.

Batch evaluation

To evaluate multiple queries in a single call, pass a list of JSON-serialized inputs. Each element represents one retrieval evaluation:

results = evaluate(
    "hit_rate",
    hypothesis=[
        json.dumps(["Paris is the capital of France.", "France is in Europe.", "Napoleon was born in Corsica."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["Completely unrelated.", "Nothing matches."]),
    ],
    reference=[
        json.dumps(["Paris is the capital of France.", "The Eiffel Tower was built in 1889."]),
        json.dumps(["The sky is blue.", "Water is wet."]),
        json.dumps(["The Louvre is in Paris."]),
    ],
)

for i, r in enumerate(results):
    print(f"Query {i+1}: {r.score}")
# Query 1: 1.0   (match found)
# Query 2: 1.0   (match found)
# Query 3: 0.0   (no match)

How it works

Hit Rate is the simplest retrieval metric: did the retriever find at least one relevant chunk?

Formula:

Hit Rate = 1.0  if any retrieved chunk matches a ground-truth chunk
         = 0.0  otherwise

Matching is based on exact string equality. Hit Rate is useful as a baseline sanity check. If hit rate is 0.0, the retriever completely failed to find any relevant context, and all downstream metrics (Recall, Precision, NDCG) will also be 0.

When to use

Run Hit Rate wherever you need a quick, coarse signal on whether retrieval is working at all.

• RAG and retrieval pipelines, as a first sanity check before digging into Recall@K, Precision@K, or NDCG@K
• Monitoring retrieval health over time, since a drop to 0.0 flags a broken retriever immediately
• Comparing indexing or chunking changes at a glance before running finer-grained metrics

What to do when Hit Rate fails

If hit rate is low, the retriever is completely failing to find relevant content for some queries:

• Check if the failing queries use different vocabulary or phrasing than what appears in the indexed documents
• Verify that the relevant documents are actually indexed and not filtered out during preprocessing
• For domain-specific queries, consider fine-tuning the embedding model or adding synonyms to the index
• Ensure document chunking doesn’t split relevant information into fragments too small to match
• Try hybrid retrieval (dense + sparse) to catch queries where one method fails