Summary Quality

Evaluates whether a summary effectively captures the main points, maintains factual accuracy, and achieves an appropriate length while preserving the original meaning. It checks for both the inclusion of key information and the exclusion of unnecessary details. 

result = evaluator.evaluate(
    eval_templates="summary_quality",
    inputs={
        "output": "Example output summary text",
        "input": "Example input text"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "summary_quality",
  {
    output: "Example output summary text",
    input: "Example input text"
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
outputstringThe generated summary.
inputstringThe original document or source content.
Output
FieldDescription
ResultReturns a score representing the summary quality, where higher values indicate better summary quality.
ReasonProvides a detailed explanation of the summary quality assessment.

What to Do When Summary Quality Evaluation Gives a Low Score

When a summary quality evaluation yields a low score, the first step is to review the evaluation criteria to ensure they are clearly defined and aligned with the assessment goals. If necessary, adjustments should be made to enhance their comprehensiveness and relevance.

Next, the summary itself should be analysed for completeness, accuracy, and relevance, identifying any gaps or inaccuracies. Refinements should be considered to better capture the main points and improve the overall quality of the summary.

Was this page helpful?

Questions & Discussion