Numeric Similarity
Calculates the normalized numerical difference between a generated value and a reference value, extracted from generated output.
Numeric Similarity checks whether a numeric value in the generated output matches a reference value. Run it when the correctness of a specific number matters more than the surrounding wording.
What it does
Numeric Similarity is a statistical metric. It extracts numeric values from the output and the expected content, then computes the absolute or normalized difference between them.
Input
| Required Input | Type | Description |
|---|---|---|
expected | string | Reference content with the expected numeric value |
output | string | Model-generated content containing the numeric prediction |
Output
| Field | Type | Description |
|---|---|---|
| Result | score | Represents the normalized difference between the numeric values, where higher values indicate greater similarity |
| Reason | string | A plain-language explanation of the numeric similarity assessment |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"numeric_similarity",
expected="The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output="The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high.",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"numeric_similarity",
{
expected: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
output: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Numeric Similarity wherever the answer hinges on a specific number and lexical or semantic metrics could overlook an outright numeric error.
- Text outputs that report measurements, prices, counts, or dates
- Generated content where numeric correctness needs to be checked explicitly, separate from wording
- Regression checks on numeric fields that a semantic metric would otherwise score as similar despite a wrong value
What to do when Numeric Similarity is low
Check that the output actually contains an extractable numeric value in a recognizable format; unit mismatches or embedded text around the number can throw off extraction. Confirm the reference value is stated in the same unit and scale as the expected output.
If the model is consistently off by a fixed factor or rounding differently, adjust the prompt or post-processing to normalize units before comparison.
Questions & Discussion