Embedding Similarity

Measures semantic similarity between the generated and reference content. 

result = evaluator.evaluate(
    eval_templates="embedding_similarity",
    inputs={
        "expected": "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
        "output": "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "embedding_similarity",
  {
    expected: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
    output: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
expectedstringReference content for comparison against the model generated output
outputstringModel-generated output to be evaluated for embedding similarity
Output
FieldDescription
ResultReturns score, where higher score indicates stronger similarity
ReasonProvides a detailed explanation of the embedding similarity assessment

About Embedding Similarity

It evaluates how similar two texts are in meaning by comparing their vector embeddings using distance-based similarity measures. Traditional metrics like BLEU or ROUGE rely on word overlap and can fail when the generated output is a valid paraphrase with no lexical match.

How Similarity Is Calculated?

Once both texts are encoded into a high-dimensional vector representations, the similarity between the two vectors u and v is computed using one of the following methods:

  1. Cosine Similarity: Measures the cosine of the angle between vectors.

Cosine Similarity = 1 - u × v||u|| ||v||

  1. Euclidean Distance: Measures the straight-line distance between vectors (L2 Norm).

    Euclidean Distance = √( Σ_i=1)^(n (u_i - v_i)^2 )

  2. Manhattan Distance: Measures sum of absolute differences between vectors (L1 Norm).

Manhattan Distance = Σ |u_i - v_i|

Was this page helpful?

Questions & Discussion