Definition

Evaluates whether a summary effectively captures the main points, maintains factual accuracy, and achieves an appropriate length while preserving the original meaning. It checks for both the inclusion of key information and the exclusion of unnecessary details.

Calculation

The evaluation process begins with configuration setup, where the input, output, and context are defined, along with the evaluation criteria that guide the assessment. In the quality analysis phase, the summary is evaluated for its ability to concisely capture the main points while maintaining accuracy and relevance to the original content.

The assessment determines whether the summary includes all necessary information while excluding irrelevant details.

Eval returns a score based on the quality analysis, which is then compared against predefined criteria to determine whether the summary meets the expected standards.

What to Do When Summary Quality Evaluation Gives a Low Score

When a summary quality evaluation yields a low score, the first step is to review the evaluation criteria to ensure they are clearly defined and aligned with the assessment goals. If necessary, adjustments should be made to enhance their comprehensiveness and relevance.

Next, the summary itself should be analysed for completeness, accuracy, and relevance, identifying any gaps or inaccuracies. Refinements should be considered to better capture the main points and improve the overall quality of the summary.

Differentiating Summary Quality with Summarization Accuracy

Summarization Accuracy focuses specifically on the factual correctness of a summary compared to the original document, ensuring that no information is misrepresented. While Summary Quality takes a broader approach, evaluating the overall effectiveness of a summary, including completeness, relevance, and conciseness.

In terms of inputs, Summarization Accuracy requires only a document and a summary, whereas Summary Quality also incorporates input, output, and context for a more comprehensive assessment.

The output of Summarization Accuracy provides a Pass/Fail result based on factual correctness, while Summary Quality assigns a score reflecting various quality metrics.