Chunk Utilization

Measures how effectively a language model leverages information from the provided context to produce a coherent and contextually appropriate output. 

result = evaluator.evaluate(
    eval_templates="chunk_utilization",
    inputs={
        "context": [
            "Paris is the capital and largest city of France.",
            "France is a country in Western Europe.",
            "Paris is known for its art museums and fashion districts."
        ],
        "output": "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
        "input": "What is the capital of France?"
    },
    model_name="turing_flash"
)

print(result.eval_results[0].output)
print(result.eval_results[0].reason)
import { Evaluator, Templates } from "@future-agi/ai-evaluation";

const evaluator = new Evaluator();

const result = await evaluator.evaluate(
  "chunk_utilization",
  {
    context: [
      "Paris is the capital and largest city of France.",
      "France is a country in Western Europe.",
      "Paris is known for its art museums and fashion districts."
    ],
    output: "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
    input: "What is the capital of France?"
  },
  {
    modelName: "turing_flash",
  }
);

console.log(result);
Input
Required InputTypeDescription
contextstring or list[string]The contextual information provided to the model
outputstringThe response generated by the language model
Output
FieldDescription
ResultReturns a numeric score, where higher values indicate more effective utilization of context
ReasonProvides a detailed explanation of the evaluation

What to Do When Chunk Utilization Score is Low

  • Ensure that the context provided is relevant and sufficiently detailed for the model to utilise effectively.
  • Modify the input prompt to better guide the model in using the context. Clearer instructions may help the model understand how to incorporate the context into its response.
  • If the model consistently fails to use context, it may require retraining or fine-tuning with more examples that emphasise the importance of context utilization.

Comparing Chunk Utilization with Similar Evals

  • Chunk Attribution: Chunk Attribution assesses whether the model acknowledges and references the provided context at all (Pass/Fail), while Chunk Utilization evaluates how effectively the model incorporates that context into its response (numeric score).
Was this page helpful?

Questions & Discussion