Definition

Provides a ranking score for each context based on specified criteria. This evaluation ensures that contexts are ranked according to their relevance and suitability for the given input.

A high score indicates a context that is highly relevant and suitable, while a low score suggests less relevance.

Calculation

The evaluation process begins with configuring the input and contexts to be assessed, along with defining the ranking criteria that guide the evaluation. Ranking analysis is then performed to evaluate the relevance and suitability of each context in relation to the input, assigning a ranking score based on the specified criteria. Finally, a ranking score is assigned to each context, and the scores are compared to determine their relative ranking.

What to do if the Eval Ranking is Low

If the evaluation returns a low ranking score, the ranking criteria should be reviewed to ensure they are well-defined, relevant, and aligned with the evaluation’s objectives. Adjustments may be necessary to enhance clarity and comprehensiveness. Additionally, the contexts should be analysed for relevance and suitability, identifying any gaps or inadequacies and refining them as needed to better support the input.

Differentiating Eval Ranking with Context Adherence

Eval Ranking and Context Adherence serve distinct purposes. Eval Ranking focuses on ranking contexts based on their relevance and suitability for the input, ensuring that the most appropriate context is identified. In contrast, Context Adherence evaluates how well a response stays within the provided context, ensuring that no external information is introduced.