Skip to main content

​
Overview

It evaluates how similar two texts are in meaning by comparing their vector embeddings using distance-based similarity measures. Traditional metrics like BLEU or ROUGE rely on word overlap and can fail when the generated output is a valid paraphrase with no lexical match.

​
How Similarity Is Calculated?

Once both texts are encoded into a high-dimensional vector representations, the similarity between the two vectors u and v is computed using one of the following methods:
  1. Cosine Similarity: Measures the cosine of the angle between vectors.
Cosine Similarity=1−u⋅v∥u∥∥v∥\text{Cosine Similarity} = 1 - \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}
  1. Euclidean Distance: Measures the straight-line distance between vectors (L2 Norm). Euclidean Distance=∑i=1n(ui−vi)2\text{Euclidean Distance} = \sqrt{ \sum_{i=1}^{n} (u_i - v_i)^2 }
  2. Manhattan Distance: Measures sum of absolute differences between vectors (L1 Norm).
Manhattan Distance=∑i=1n∣ui−vi∣\text{Manhattan Distance} = {\sum_{i=1}^{n} |u_i - v_i|}

​
Evaluation Using SDK

Click here to learn how to setup evaluation using SDK.
Input & Configuration:
ParameterTypeDescription
Required InputsresponsestrModel-generated output to be evaluated.
expected_textstr or List[str]One or more reference texts for comparison.
Optional Configsimilarity_methodstrDistance function used to compare embedding vectors. Options: "cosine" (default), "euclidean", "manhattan".
normalizeboolWhether to normalize embedding vectors before computing similarity. Default is True.
Parameter Options:
Parameter - similarity_methodDescription
cosineMeasures the cosine of the angle between two vectors
euclideanComputes the straight-line (L2) distance between vectors
manhattanComputes the L1 (absolute) distance between vectors
Output:
Output FieldTypeDescription
scorefloatValue between 0 and 1 representing semantic similarity. Higher values indicate stronger similarity.
result = evaluator.evaluate(
    eval_templates="embedding_similarity",
    inputs={
        "expected_text": "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
        "response": "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)