Context Relevance

Evaluates whether the provided context is sufficient and relevant to answer the query, surfacing weak RAG retrieval.

Context Relevance checks whether the context retrieved for a query is actually relevant and sufficient to answer it. Run it to catch weak or off-target retrieval before it produces a bad response.

What it does

Context Relevance is an LLM-as-Judge eval. It reads the input query and the retrieved context, then scores how relevant and sufficient that context is for answering the query.

Input

Required Input	Type	Description
`context`	`string`	The context provided to the model
`input`	`string`	The input provided to the model

Output

Field	Type	Description
Result	`score`	Higher scores indicate more relevant context
Reason	`string`	A plain-language explanation of the context relevance assessment

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "context_relevance",
    context="Honey never spoils because it has low moisture content and high acidity, creating an environment that resists bacteria and microorganisms. Archaeologists have even found pots of honey in ancient Egyptian tombs that are still perfectly edible.",
    input="Why doesn't honey go bad?",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "context_relevance",
  {
    context: "Honey never spoils because it has low moisture content and high acidity, creating an environment that resists bacteria and microorganisms. Archaeologists have even found pots of honey in ancient Egyptian tombs that are still perfectly edible.",
    input: "Why doesn't honey go bad?"
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Context Relevance wherever a query drives context retrieval and you need to confirm the retrieved material actually supports an answer.

Text, audio, image, and chat outputs where context is retrieved before generation
RAG and retrieval pipelines, to check whether retrieval surfaces chunks that support the query

What to do when Context Relevance fails

When context relevance is low, the first step is to identify which parts of the context are either irrelevant or insufficient to address the query effectively.

If critical information is missing, additional details should be incorporated to ensure completeness. At the same time, any irrelevant content should be removed or refined to improve focus and alignment with the query.

Implementing mechanisms to enhance context-query alignment can further strengthen relevance, ensuring that only pertinent information is considered. Additionally, optimising context retrieval processes can help prioritise relevant details, improving overall response accuracy and coherence.

Was this page helpful?

Questions & Discussion