Definition

Evaluates whether the provided context is sufficient and relevant to answer the given input query. This evaluation is crucial for RAG systems to ensure that retrieved context pieces contain the necessary information to generate accurate responses.

Calculation

The evaluation process begins with the system tokenising the provided context, breaking it into manageable units for processing. It then maps key concepts and relationships while assigning relevance scores to different sections based on their importance to the query. In the query analysis stage, the system determines the main intent behind the query and identifies the specific information required to generate an accurate response. The relevance assessment phase involves cross-referencing the query’s information needs against the context, evaluating coverage completeness, and performing a gap analysis to detect any missing or irrelevant details.

The eval returns relevance score that quantifies how well the context supports the query. The system checks if the provided context is sufficient to produce a complete and accurate response, assigning a numerical score from 0 to 10 to indicate the level of context relevance.

What to do when Context Relevance is Low

When context relevance is low, the first step is to identify which parts of the context are either irrelevant or insufficient to address the query effectively.

If critical information is missing, additional details should be incorporated to ensure completeness. At the same time, any irrelevant content should be removed or refined to improve focus and alignment with the query.

Implementing mechanisms to enhance context-query alignment can further strengthen relevance, ensuring that only pertinent information is considered. Additionally, optimising context retrieval processes can help prioritise relevant details, improving overall response accuracy and coherence.

Differentiating Context Relevance with Similar Evals

  1. Context Adherence: It measures how well responses stay within the provided context while Context Relevance evaluates the sufficiency and appropriateness of the context.
  2. Completeness: Completeness evaluates if the response completely answers the query, while Context Relevance focuses on the context’s ability to support a complete response.
  3. Context Similarity: It compares similarity between provided and expected context, that is, it measures how closely the context matches expected information, while Context Relevance assesses if the context is sufficient and appropriate for the query.