Eval Ranking
Assigns a ranking score to each context based on specified criteria, ordering retrieved contexts by relevance to the input.
Eval Ranking scores each candidate context against a query so you can see which one is the best fit, not just whether context passed a relevance bar. Run it when you need to rank, not just check, retrieved context.
What it does
Eval Ranking is an LLM-as-Ranker eval. It reads the input query and a list of candidate contexts, then scores each context’s ranking quality for that query.
Input
| Required Input | Type | Description |
|---|---|---|
input | string | The input provided to the model |
context | list[string] | List of contexts to rank |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher values indicate better ranking quality of that context |
| Reason | string | A plain-language explanation of the ranking assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"eval_ranking",
input="What is the solar system?",
context=[
"The solar system consists of the Sun and celestial objects bound to it",
"Our solar system formed 4.6 billion years ago"
],
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"eval_ranking",
{
input: "What is the solar system?",
context: [
"The solar system consists of the Sun and celestial objects bound to it",
"Our solar system formed 4.6 billion years ago"
]
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Eval Ranking wherever you have multiple candidate contexts and need to know which ones are the best match, not just a pass/fail relevance check.
- RAG and retrieval pipelines, to order retrieved chunks by relevance and suitability before generation
- Custom retrieval evaluation, where ranking criteria are specific to your domain
What to do when Eval Ranking fails
If the evaluation returns a low ranking score, review the ranking criteria to ensure they’re well-defined, relevant, and aligned with the evaluation’s objectives, adjusting them for clarity and comprehensiveness where needed.
Analyze the contexts themselves for relevance and suitability, identifying any gaps or inadequacies and refining them to better support the input.
Questions & Discussion