Summaries play a critical role in distilling large amounts of information into concise, digestible content. Whether used in business reports, research articles, legal documents, or AI-generated content, summaries must maintain both clarity and accuracy while effectively representing the original document. However, poorly generated summaries can lead to:
  • Loss of critical details – Important information may be omitted.
  • Misinterpretations – The summary may introduce errors that distort the original meaning.
  • Lack of relevance – Unnecessary or unrelated information may be included.
To ensure high-quality summaries, we will use Summary Quality eval: Both evaluations work together to verify that summaries are concise, accurate, and representative of the original content, reducing the risk of misinformation or misleading conclusions.

​
Summary Quality

Evaluates if a summary effectively captures the main points, maintains factual accuracy, and achieves an appropriate length while preserving the original meaning. Ensures key information is included while unnecessary details are excluded. Click here to read the eval definition of Summary Quality

​
a. Using Interface

​
Required Inputs

  • output: The generated summary.

​
Optional Inputs

  • context: Additional background information (if applicable).
  • input: The original document or source content.

​
Config

  • Check Internet: Whether to verify information using external sources.

​
Output

Returns a float between 0 and 1, where higher values indicate better summary quality.

​
b. Using SDK

Export your API key and Secret key into your environment variables. 
result = evaluator.evaluate(
    eval_templates="summary_quality",
    inputs={
        "context": "The Amazon rainforest, often referred to as the 'lungs of the Earth', produces 20% of the world's oxygen. It is home to an estimated 390 billion individual trees, representing around 16,000 different species. However, deforestation has become a major threat, with large areas being cleared every year for agriculture and development.",
        "input": "Summarize article on Amazon rainforest given in context",
        "output": "The Amazon rainforest is a vital source of global oxygen and biodiversity, but faces serious threats from ongoing deforestation."
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)