​
Evaluation Using Interface

Input:
  • Required Inputs:
    • output: The generated summary column.
  • Optional Inputs:
    • context: Additional background information column (if applicable).
    • input: The original document or source content column.
  • Configuration Parameters:
    • Check Internet: Boolean - Whether to verify information using external sources.
Output:
  • Score: Percentage score between 0 and 100
Interpretation:
  • Higher scores: Indicate a better quality summary that effectively captures main points, is factually accurate, appropriately concise, and preserves original meaning.
  • Lower scores: Suggest issues with the summary, such as missing key information, factual inaccuracies, inclusion of unnecessary details, or poor conciseness.

​
Evaluation Using Python SDK

Click here to learn how to setup evaluation using the Python SDK.
Input TypeParameterTypeDescription
Required InputsoutputstringThe generated summary.
Optional InputscontextstringAdditional background information (if applicable).
inputstringThe original document or source content.
Configuration Parameterscheck_internetboolWhether to verify information using external sources.
OutputTypeDescription
ScorefloatReturns a score between 0 and 1, where higher values indicate better summary quality.
from fi.evals import Evaluator
from fi.testcases import TestCase
from fi.evals.templates import SummaryQuality

summary_quality = SummaryQuality(config={"check_internet": False})

test_case = TestCase(
		output= "Example output summary text",
    context="Example context text",
    input="Example input text"
)

result = evaluator.evaluate(eval_templates=[summary_quality], inputs=[test_case], model_name="turing_flash")
accuracy_score = result.eval_results[0].metrics[0].value

​
What to Do When Summary Quality Evaluation Gives a Low Score

When a summary quality evaluation yields a low score, the first step is to review the evaluation criteria to ensure they are clearly defined and aligned with the assessment goals. If necessary, adjustments should be made to enhance their comprehensiveness and relevance. Next, the summary itself should be analysed for completeness, accuracy, and relevance, identifying any gaps or inaccuracies. Refinements should be considered to better capture the main points and improve the overall quality of the summary.