Levenshtein Similarity
Calculates edit distance between generated and reference text, scoring similarity by the minimum number of single-character edits required.
Levenshtein Similarity measures how close two texts are by counting the character-level edits needed to turn one into the other. Run it when you need a strict, deterministic comparison rather than a meaning-based one.
What it does
Levenshtein Similarity is a statistical metric. It reads the output and the expected content, then computes the minimum number of insertions, deletions, and substitutions needed to transform one string into the other, normalized to a score between 0 and 1.
Input
| Required Input | Type | Description |
|---|---|---|
expected | string | Reference content for comparison against the model generated output |
output | string | Model generated content to be evaluated for similarity |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Higher score indicates greater similarity |
| Reason | string | A plain-language explanation of the similarity assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"levenshtein_similarity",
expected="The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output="The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"levenshtein_similarity",
{
expected: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Levenshtein Similarity wherever exact character-level fidelity matters more than semantic equivalence.
- Text outputs from spelling correction or OCR, where character-level accuracy is the point
- Deterministic text matching against a fixed reference string
- Regression checks where you want to flag any character drift, not just meaning changes
What to do when Levenshtein Similarity is low
Consider case sensitivity, since the comparison is typically case-sensitive. Check for whitespace and punctuation differences, which count as edits.
For meaning-based comparison rather than exact character matching, consider semantic similarity metrics. For texts with similar meaning but different wording, consider ROUGE, BLEU, or Embedding Similarity instead. Remember that this metric measures character-level similarity, not semantic similarity.
Questions & Discussion