Summary Quality

Evaluates if a summary captures the main points of the source content and achieves an appropriate length.

Summary Quality checks whether a generated summary captures the source content’s main points at an appropriate length. Run it wherever a model is condensing longer content into a shorter form.

What it does

Summary Quality is an LLM-as-Judge eval. It reads the original content and the generated summary, then scores how well the summary captures the source.

Input

Required Input	Type	Description
`output`	`string`	The generated summary
`input`	`string`	The original document or source content

Output

Field	Type	Description
Result	`score`	Higher values indicate better summary quality
Reason	`string`	A plain-language explanation of the summary quality assessment

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "summary_quality",
    output="Example output summary text",
    input="Example input text",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "summary_quality",
  {
    output: "Example output summary text",
    input: "Example input text"
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Summary Quality wherever a model condenses source content and length or coverage matters.

Text, audio, and image content summarized from a longer source
RAG and retrieval pipelines, to check summaries built from retrieved documents
Document processing, where a summary needs to hit both key points and target length

What to do when Summary Quality fails

When summary quality scores low, start by reviewing the evaluation criteria to make sure they’re clearly defined and aligned with the assessment goals. Adjust them if they need to be more comprehensive or relevant.

Next, analyze the summary itself for completeness, accuracy, and relevance, and identify any gaps or inaccuracies. Refine the summary to better capture the main points and improve its overall quality.

Was this page helpful?

Questions & Discussion