Embedding Similarity
Calculates semantic similarity between generated and reference text using embedding vectors, capturing meaning beyond surface wording.
Embedding Similarity checks how close two texts are in meaning, even when they share little or no vocabulary. Run it when a valid paraphrase should score well but lexical metrics like BLEU or ROUGE would miss it.
What it does
Embedding Similarity is a statistical metric. It encodes the output and the expected content into vector embeddings, then computes a distance-based similarity between the two vectors using cosine similarity, Euclidean distance, or Manhattan distance.
Input
| Required Input | Type | Description |
|---|---|---|
expected | string | Reference content for comparison against the model generated output |
output | string | Model-generated output to be evaluated for embedding similarity |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher scores indicate stronger similarity |
| Reason | string | A plain-language explanation of the score |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"embedding_similarity",
expected="The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output="The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"embedding_similarity",
{
expected: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Embedding Similarity wherever the generated text is allowed to paraphrase and you care about meaning rather than exact wording.
- Text outputs where a correct answer can be phrased many different ways
- RAG & Retrieval comparisons where lexical overlap metrics would under-score valid paraphrases
- Cases where BLEU or ROUGE score a semantically correct answer too low because of low word overlap
What to do when Embedding Similarity is low
Check whether the output and expected content are actually about the same subject; a low score can mean the response drifted off-topic rather than just being worded differently. Review which distance measure is configured, since cosine similarity, Euclidean distance, and Manhattan distance can rank the same pair differently.
If short texts or single words are being compared, results can be less reliable since embeddings are trained on richer context. For a stricter, non-semantic check, pair this with Levenshtein Similarity or Fuzzy Match.
Questions & Discussion