Retrieval Metrics

Code-based metrics that score retrieval and ranking quality against reference relevant items, no LLM judge.

These are code-based (CustomCodeEval) metrics for retrieval and RAG pipelines: each one compares retrieved contexts or ranked items against a reference set using exact string matching, then returns a normalized 0-1 score, no LLM judge involved.

Metrics

Metric	What it measures	Required inputs	Output
`non_llm_context_precision`	Fraction of retrieved contexts that exact-match a reference context	`output`, `expected`	score (0-1), higher = better
`non_llm_context_recall`	Fraction of reference contexts that were successfully retrieved	`output`, `expected`	score (0-1), higher = better
`mean_average_precision`	Average precision at each relevant rank position, rewards relevant items retrieved earlier	`reference`, `hypothesis`	score (0-1), higher = better

Run a metric from code

Call evaluate() with the template id and the metric’s required inputs. Swap the template id to run any metric in this table.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "mean_average_precision",
    hypothesis=["doc_3", "doc_1", "doc_9", "doc_4"],
    reference=["doc_1", "doc_4", "doc_7"],
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "mean_average_precision",
  {
    hypothesis: ["doc_3", "doc_1", "doc_9", "doc_4"],
    reference: ["doc_1", "doc_4", "doc_7"],
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Reach for these when you’re scoring a retriever or RAG index offline against known-relevant items and don’t need an LLM judge.

Evaluating a retriever or vector index against a labeled set of relevant contexts, without calling a judge model
Comparing ranking configurations (chunking, embedding model, top-k) using mean_average_precision, which credits relevant items retrieved earlier
Tracking precision and recall of retrieved chunks against ground truth as a RAG pipeline changes over time

Was this page helpful?

Questions & Discussion