Eval Definition
Is Factually Consistent
Evaluates whether output content is factually consistent with provided input or context
Evaluation Using Interface
Input:
- Required Inputs:
- input: The source material, context, or question.
- output: The response to be evaluated for factual consistency.
Output:
- Result: Returns ‘Passed’ if the output is factually consistent with the input, ‘Failed’ if it contains inconsistencies.
Evaluation Using Python SDK
Click here to learn how to setup evaluation using the Python SDK.
Input:
- Required Inputs:
- input:
string
- The source material, context, or question. - output:
string
- The response to be evaluated for factual consistency.
- input:
Output:
- Result: Returns a list containing ‘Passed’ if the output is factually consistent with the input, or ‘Failed’ if it contains inconsistencies.
- Reason: Provides a detailed explanation of why the response was deemed factually consistent or inconsistent.
Example Output:
What to do If you get Undesired Results
If the content is evaluated as factually inconsistent (Failed) and you want to improve it:
- Verify all facts against reliable sources or the provided context
- Remove any claims or details not supported by the source material
- Correct any inaccuracies, contradictions, or misrepresentations
- Ensure numbers, dates, names, and specific details align with the source
- Avoid extrapolating beyond what is explicitly stated in the source
- Use qualifying language (like “may,” “could,” or “suggests”) when appropriate
- Cite specific parts of the source material when providing information
Comparing Is Factually Consistent with Similar Evals
- Factual Accuracy: While Is Factually Consistent focuses on consistency with the provided input or context, Factual Accuracy might verify claims against broader world knowledge.
- Groundedness: Is Factually Consistent evaluates whether output contradicts the source, while Groundedness measures how well the output is supported by the source.