Summary Quality
Evaluates whether a summary effectively captures the main points, maintains factual accuracy, and achieves an appropriate length while preserving the original meaning. It checks for both the inclusion of key information and the exclusion of unnecessary details.
Evaluation Using Interface
Input:
- Required Inputs:
- output: The generated summary column.
- Optional Inputs:
- context: Additional background information column (if applicable).
- input: The original document or source content column.
- Configuration Parameters:
- Check Internet: Boolean - Whether to verify information using external sources.
Output:
- Score: Percentage score between 0 and 100
Interpretation:
- Higher scores: Indicate a better quality summary that effectively captures main points, is factually accurate, appropriately concise, and preserves original meaning.
- Lower scores: Suggest issues with the summary, such as missing key information, factual inaccuracies, inclusion of unnecessary details, or poor conciseness.
Evaluation Using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input Type | Parameter | Type | Description |
---|---|---|---|
Required Inputs | output | string | The generated summary. |
Optional Inputs | context | string | Additional background information (if applicable). |
input | string | The original document or source content. | |
Configuration Parameters | check_internet | bool | Whether to verify information using external sources. |
Output | Type | Description |
---|---|---|
Score | float | Returns a score between 0 and 1, where higher values indicate better summary quality. |
What to Do When Summary Quality Evaluation Gives a Low Score
When a summary quality evaluation yields a low score, the first step is to review the evaluation criteria to ensure they are clearly defined and aligned with the assessment goals. If necessary, adjustments should be made to enhance their comprehensiveness and relevance.
Next, the summary itself should be analysed for completeness, accuracy, and relevance, identifying any gaps or inaccuracies. Refinements should be considered to better capture the main points and improve the overall quality of the summary.