ROUGE Score

Calculates recall-oriented n-gram overlap between generated and reference text, commonly used to evaluate summarization quality.

ROUGE Score measures how much of a reference text’s content is recovered in the generated text. Run it when recall against a reference matters as much as, or more than, precision.

What it does

ROUGE Score is a statistical metric. It reads the hypothesis and the reference text, then measures overlapping n-grams and reports them as an F1-score, the harmonic mean of precision and recall.

Input

Required Input	Type	Description
`reference`	`string`	The reference containing the information to be captured
`hypothesis`	`string`	The content to be evaluated for recall against the reference

Output

Field	Type	Description
Result	`score`	Higher values indicate better recall of the hypothesis against the reference
Reason	`string`	A plain-language explanation of the recall evaluation

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "rouge_score",
    reference="The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
    hypothesis="The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high.",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "rouge_score",
  {
    reference: "The Eiffel Tower is a famous landmark in Paris, built in 1889 for the World's Fair. It stands 324 meters tall.",
    hypothesis: "The Eiffel Tower, located in Paris, was built in 1889 and is 324 meters high."
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run ROUGE Score wherever recovering the source’s key content matters more than matching its exact phrasing.

Summarization tasks, to check whether the model covered the important parts of the source
Cases where structure and ordering matter but exact phrasing can vary, using ROUGE-L
Translation quality checks where recall of reference content is the priority

What to do when ROUGE Score is low

Use "rougeL" if the phrasing of the generated text differs from the reference but the meaning is preserved, since it credits the longest common subsequence rather than fixed n-grams. Apply use_stemmer=True to improve robustness to word-form variation.

Aggregate the ROUGE score with semantic evals like Embedding Similarity using an Aggregated Metric for a more holistic comparison of generated text against the reference.

Was this page helpful?

Questions & Discussion