BLEU Score

BLEU Score measures how many contiguous word sequences (n-grams) in the generated text also appear in the reference text. Run it when you need a fast, established lexical-overlap metric for translation or summarization output.

What it does

BLEU Score is a statistical metric. It reads the hypothesis and the reference text, then computes a modified n-gram precision, combined across n-gram levels and adjusted by a brevity penalty for outputs shorter than the reference.

Input

Output

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

from fi.evals import evaluate

result = evaluate(
    "bleu_score",
    reference="The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
    hypothesis="The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high.",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "bleu_score",
  {
    reference: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
    hypothesis: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run BLEU Score wherever you need a fast, reproducible lexical-overlap check against a fixed reference text.

• Machine translation output, comparing a translated hypothesis against a reference translation
• Summarization quality checks where word choice is expected to closely track the source
• Regression testing on text generation, to catch large lexical drift between runs

What to do when BLEU Score is low

A low score can come from the model generating correct meaning with different word ordering, which reduces higher-order n-gram precision; try reducing max_n_gram in that case. It can also come from the generated text being shorter than the reference, which triggers the brevity penalty.

Aggregate the BLEU score with semantic evals like Embedding Similarity using an Aggregated Metric for a more holistic comparison. You can also switch smoothing method to mitigate the impact of zero matches at higher n-gram levels: