Eval Ranking
Provides a ranking score for each context based on specified criteria. This evaluation ensures that contexts are ranked according to their relevance and suitability for the given input.
Evaluation Using Interface
Input:
- Required Inputs:
- input: The query column.
- context: The list of context columns to rank.
- Configuration Parameters:
- criteria: Ranking criteria description (e.g., “Rank contexts based on relevance to the query”).
Output:
- Score: Percentage score between 0 and 100
Interpretation:
- Higher scores: Indicate that the contexts are ranked effectively according to the specified criteria, with the most relevant contexts ranked higher.
- Lower scores: Suggest poor ranking quality, where less relevant contexts might be ranked higher than more relevant ones.
Evaluation Using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input Type | Parameter | Type | Description |
---|---|---|---|
Required Inputs | input | string | The query. |
context | list[string] | List of contexts to rank. | |
Configuration Parameters | criteria | string | Ranking criteria description. |
Output | Type | Description |
---|---|---|
Score | float | Returns a score between 0 and 1, where higher values indicate better ranking quality. |
What to do if the Eval Ranking is Low
If the evaluation returns a low ranking score, the ranking criteria should be reviewed to ensure they are well-defined, relevant, and aligned with the evaluation’s objectives. Adjustments may be necessary to enhance clarity and comprehensiveness. Additionally, the contexts should be analysed for relevance and suitability, identifying any gaps or inadequacies and refining them as needed to better support the input.
Differentiating Eval Ranking with Context Adherence
Eval Ranking and Context Adherence serve distinct purposes. Eval Ranking focuses on ranking contexts based on their relevance and suitability for the input, ensuring that the most appropriate context is identified. In contrast, Context Adherence evaluates how well a response stays within the provided context, ensuring that no external information is introduced.
Was this page helpful?