Is Good Summary

Evaluates whether a summary effectively captures the key information from the original source content.

Is Good Summary gives a pass or fail read on whether a summary captures the source content’s key information. Run it wherever you need a quick binary check rather than a graded score.

What it does

Is Good Summary is an LLM-as-Judge eval. It reads the original source content and the generated summary, then checks whether the summary effectively captures the key information.

Input

Required Input	Type	Description
`input`	`string`	The original source content
`output`	`string`	Generated summary by the model to be evaluated

Output

Field	Type	Description
Result	`Pass / Fail`	Pass means the summary effectively captures the key information; Fail means it doesn’t
Reason	`string`	A plain-language explanation of why the summary was deemed good or poor

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "is_good_summary",
    input="Honey never spoils because it has low moisture content and high acidity, creating an environment that resists bacteria and microorganisms. Archaeologists have even found pots of honey in ancient Egyptian tombs that are still perfectly edible.",
    output="Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes.",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "is_good_summary",
  {
    input: "Honey never spoils because it has low moisture content and high acidity, creating an environment that resists bacteria and microorganisms. Archaeologists have even found pots of honey in ancient Egyptian tombs that are still perfectly edible.",
    output: "Honey doesn't spoil because its low moisture and high acidity prevent the growth of bacteria and other microbes."
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Is Good Summary wherever a summary needs a straightforward pass or fail signal.

Text and audio content summarized from a longer source
RAG and retrieval pipelines, to spot-check summaries built from retrieved documents
Quick quality gates, where a binary result is easier to act on than a graded score

What to do when Is Good Summary fails

If the summary is evaluated as poor, ensure all key points from the original source content are included and the core meaning and intent are maintained. Remove unnecessary details but keep essential information, and keep the summary concise while preserving important context.

Avoid adding new information not present in the original source content, and use clear language that accurately represents it.

Was this page helpful?

Questions & Discussion