Definition

Context adherence evaluates how well responses stay within the provided context by measuring if the output contains any information not present in the given context. This evaluation is crucial for ensuring factual consistency and preventing hallucination in responses.

Calculation

The evaluation process systematically compares the generated output against the provided context to assess its adherence and factual consistency. It requires an output response for evaluation, a reference context, and optionally, an internet check for external verification.

The process begins with context analysis, where the system tokenises and indexes the reference context, creating a structured knowledge base that maps key concepts and relationships. Next, during output verification, the generated response is analysed to identify statements and claims, which are then cross-referenced with the contextual knowledge base to determine their validity.

Scoring is performed by measuring the percentage of statements supported by the context, detecting unsupported information, evaluating the relevance of used context, and ensuring factual consistency. The evaluation produces a numerical score ranging from 0 to 10, with higher scores indicating stronger adherence to the context. Additionally, the system provides explanations for any detected deviations.

What to do when Context Adherence is Low

When context adherence is low, start by identifying statements that are not supported by the provided context and checking for implicit versus explicit information to assess potential misinterpretations.

Reviewing how the context is processed can help pinpoint inconsistencies. If necessary, expand context coverage to fill in gaps, clarify ambiguous details, and add missing relevant information.

To improve adherence, implement stricter context binding, integrate fact-checking mechanisms, and enhance overall context processing.

Comparing Context Adherence with Similar Evals

  1. Context Relevance: While Context Adherence focuses on staying within context bounds, Context Relevance evaluates if the provided context is sufficient and appropriate for the query.
  2. Prompt/Instruction Adherence: Context Adherence measures factual consistency with context, while Prompt Adherence evaluates following instructions and format requirements.