Skip to main content
result = evaluator.evaluate(
    eval_templates="chunk_utilization",
    inputs={
        "context": [
            "Paris is the capital and largest city of France.",
            "France is a country in Western Europe.",
            "Paris is known for its art museums and fashion districts."
        ],
        "output": "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
        "input": "What is the capital of France?"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
Input
Required InputTypeDescription
contextstring or list[string]The contextual information provided to the model
outputstringThe response generated by the language model
Output
FieldDescription
ScoreReturns a numeric score, where higher values indicate more effective utilization of context
ReasonProvides a detailed explanation of the evaluation

What to Do When Chunk Utilization Score is Low

  • Ensure that the context provided is relevant and sufficiently detailed for the model to utilise effectively.
  • Modify the input prompt to better guide the model in using the context. Clearer instructions may help the model understand how to incorporate the context into its response.
  • If the model consistently fails to use context, it may require retraining or fine-tuning with more examples that emphasise the importance of context utilization.

Differentiating Chunk Utilization with Chunk Attribution

Chunk Attribution assesses whether the model acknowledges and references the provided context at all, yielding a binary result: Pass if the context is used, or Fail if it is not. In contrast, Chunk Utilization evaluates how effectively the model incorporates that context into its response, producing a score that reflects the depth of its reliance on the information. While Attribution checks if the context was used, Utilization measures how well it was used to generate a meaningful and informed output.
I