Evaluation Using Interface

Input:

  • Required Inputs:
    • expected_response: The reference answer column.
    • response: The generated answer column.
  • Configuration Parameters:
    • Comparator: The method used for comparison (e.g., Cosine, Exact Match).
    • Failure Threshold: Float (e.g., 0.7) - The similarity score below which the evaluation is considered a failure.

Output:

  • Score: Percentage score between 0 and 100

Interpretation:

  • Scores ≥ (Failure Threshold * 100): Indicate that the generated response is sufficiently similar to the expected_response based on the chosen Comparator.
  • Scores < (Failure Threshold * 100): Suggest that the response deviates significantly from the expected_response.

Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.


Input TypeParameterTypeDescription
Required Inputsexpected_responsestringThe reference answer.
responsestringThe generated answer.
Configuration ParameterscomparatorstringThe method to use for comparison (e.g., Comparator.COSINE.value).
failure_thresholdfloatThe threshold below which the evaluation fails (e.g., 0.7).
Comparator NameClass Name
Cosine SimilarityComparator.COSINE.value
Jaccard SimilarityComparator.JACCARD.value
Normalised Levenshtein SimilarityComparator.NORMALISED_LEVENSHTEIN.value
Jaro Winckler similarityComparator.JARO_WINKLER.value
Sorensen Dice similarityComparator.SORENSEN_DICE.value
OutputTypeDescription
ScorefloatReturns a score between 0 and 1. Values ≥ failure_threshold indicate sufficient similarity.
from fi.evals import EvalClient
from fi.testcases import LLMTestCase
from fi.evals.templates import AnswerSimilarity
from fi.evals.types import Comparator

similarity_eval = AnswerSimilarity(config={
    "comparator": Comparator.COSINE.value,
    "failure_threshold": 0.8
})

test_case = LLMTestCase(
    response="example response",
    expected_response="example of expected response"
)

evaluator = EvalClient()
result = evaluator.evaluate(eval_templates=[similarity_eval], inputs=[test_case])
similarity_score = result.eval_results[0].metrics[0].value


What to Do When Answer Similarity Evaluation is Low

A response review should be conducted to reassess the actual response’s alignment with the expected response and identify discrepancies. If necessary, a comparator adjustment can be made, selecting an alternative similarity measure that better captures nuanced differences in meaning.


Differentiating Answer Similarity with Context Relevance

Answer Similarity specifically measures how closely two responses align in meaning, whereas Context Sufficiency determines whether a given context provides enough information to answer a query.

From an input perspective, Answer Similarity requires both an expected and actual response for comparison, while Context Sufficiency evaluates a query against its provided context.