Similarity & Image-Quality Metrics

Code-based metrics for string/set similarity between texts and pixel-level fidelity between images.

These are code-based metrics, not LLM judges. Jaccard, Jaro-Winkler, and Hamming score how close two texts are as sets or character sequences. SSIM and PSNR score how close a generated image is to a reference image.

Metrics

Metric	What it measures	Required inputs	Output
`jaccard_similarity`	Token-set overlap: shared tokens divided by total unique tokens across output and expected	`output`, `expected`	score (0-1)
`jaro_winkler_similarity`	Character-matching string similarity with transposition counting, boosted by a common-prefix bonus	`output`, `expected`	score (0-1)
`hamming_similarity`	Matching character positions between two strings, normalized by the longer string’s length (shorter string is padded)	`output`, `expected`	score (0-1)
`ssim`	Structural Similarity Index between two images: luminance, contrast, and structure compared on grayscale pixel data	`output`, `expected` (images)	score (0-1), higher is more similar
`psnr`	Peak Signal-to-Noise Ratio between two images, from mean squared error over RGB pixels	`output`, `expected` (images)	score (0-1), PSNR in dB divided by 50 and clamped

Run a metric from code

Call evaluate() with the template id and the metric’s required inputs. Swap the template id to run any metric in this table.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "jaccard_similarity",
    output="The quick brown fox jumps over the lazy dog",
    expected="A quick brown fox jumped over a lazy dog",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "jaccard_similarity",
  {
    output: "The quick brown fox jumps over the lazy dog",
    expected: "A quick brown fox jumped over a lazy dog",
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Reach for these when you need a fast, deterministic similarity score instead of an LLM judge.

Token or set-level overlap checks between generated and reference text, using jaccard_similarity
Fuzzy matching on short strings like names, labels, or IDs, using jaro_winkler_similarity
Positional character comparison for fixed-format strings (codes, hashes, short tokens), using hamming_similarity
Deduplication and near-duplicate detection across text outputs
Comparing a generated image against a reference image for structural or pixel-level fidelity, using ssim and psnr

Was this page helpful?

Questions & Discussion