Ranking evaluation assesses how well a ranking model ranks or orders items to meet specified criteria. This evaluation type is particularly useful when you need to validate whether your model can effectively prioritize or sort items according to given parameters.

​
Required Parameters

ParameterDescriptionRequired
inputThe input query or criteria for rankingYes
contextList of items to be rankedYes

​
Required Configuration

The evaluation accepts the following configuration:

ParameterDescriptionRequired
criteriaThe evaluation criteria for rankingYes

​
Example

from fi.evals import Ranking, EvalClient
from fi.testcases import LLMTestCase

# Initialize the evaluation client
evaluator = EvalClient(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key"
)

# Create a test case
test_case = LLMTestCase(
    input="Rank these cities by population size from largest to smallest",
    context=[
        "Tokyo, Japan",
        "Delhi, India",
        "Shanghai, China",
        "São Paulo, Brazil"
    ]
)

# Initialize the ranking evaluator
ranking_eval = Ranking(
    config = {
        "criteria": """
            Evaluate the ranking based on:
            Accuracy of order
            Adherence to ranking criteria
            Completeness of ranking
            Consistency of ordering
        """
    }
)

# Run the evaluation
result = evaluator.evaluate(ranking_eval, test_case)
print(result) # Will return a score between 0 and 1
Output EvaluationAnswer Similarity