​
Evaluation Using Interface

Input:

  • Required Inputs:
    • input: The query column.
    • context: The list of context columns to rank.
  • Configuration Parameters:
    • criteria: Ranking criteria description (e.g., “Rank contexts based on relevance to the query”).

Output:

  • Score: Percentage score between 0 and 100

Interpretation:

  • Higher scores: Indicate that the contexts are ranked effectively according to the specified criteria, with the most relevant contexts ranked higher.
  • Lower scores: Suggest poor ranking quality, where less relevant contexts might be ranked higher than more relevant ones.

​
Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.

Input TypeParameterTypeDescription
Required InputsinputstringThe query.
contextlist[string]List of contexts to rank.
Configuration ParameterscriteriastringRanking criteria description.
OutputTypeDescription
ScorefloatReturns a score between 0 and 1, where higher values indicate better ranking quality.
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import Ranking

ranking_eval = Ranking(config={
    "criteria": "Rank contexts based on relevance to the query"
})

test_case = TestCase(
    input="What is the solar system?",
    context=[
        "The solar system consists of the Sun and celestial objects bound to it",
        "Our solar system formed 4.6 billion years ago"
    ]
)

result = evaluator.evaluate(eval_templates=[ranking_eval], inputs=[test_case], model_name="turing_flash")
ranking_score = result.eval_results[0].metrics[0].value

​
What to do if the Eval Ranking is Low

If the evaluation returns a low ranking score, the ranking criteria should be reviewed to ensure they are well-defined, relevant, and aligned with the evaluation’s objectives. Adjustments may be necessary to enhance clarity and comprehensiveness. Additionally, the contexts should be analysed for relevance and suitability, identifying any gaps or inadequacies and refining them as needed to better support the input.

​
Differentiating Eval Ranking with Context Adherence

Eval Ranking and Context Adherence serve distinct purposes. Eval Ranking focuses on ranking contexts based on their relevance and suitability for the input, ensuring that the most appropriate context is identified. In contrast, Context Adherence evaluates how well a response stays within the provided context, ensuring that no external information is introduced.