Chunk Utilization

Measures how effectively a response draws on the provided context chunks, scoring the degree of reliance on retrieved information.

Chunk Utilization scores how much of the retrieved context actually contributes to the generated response, not just whether it was touched. Run it to see how well your generator makes use of what retrieval hands it.

What it does

Chunk Utilization is an LLM-as-Judge eval. It reads the context chunks and the generated output, then scores how effectively that context was incorporated into the response.

Input

Required Input	Type	Description
`context`	`string` or `list[string]`	The contextual information provided to the model
`output`	`string`	The response generated by the language model

Output

Field	Type	Description
Result	`score`	Higher values indicate more effective utilization of context
Reason	`string`	A plain-language explanation of the evaluation

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "chunk_utilization",
    context=[
        "Paris is the capital and largest city of France.",
        "France is a country in Western Europe.",
        "Paris is known for its art museums and fashion districts."
    ],
    output="According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "chunk_utilization",
  {
    context: [
      "Paris is the capital and largest city of France.",
      "France is a country in Western Europe.",
      "Paris is known for its art museums and fashion districts."
    ],
    output: "According to the provided information, Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture."
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Chunk Utilization wherever you need to know how much of the retrieved context is actually driving the response, not just whether it was referenced.

RAG and retrieval pipelines, to measure how thoroughly the generator draws on retrieved chunks
Tuning chunking or prompt strategy, where you need a graded signal rather than a Pass/Fail
Comparing generator configurations to see which one makes better use of the same retrieved context

What to do when Chunk Utilization fails

Ensure that the context provided is relevant and sufficiently detailed for the model to utilize effectively
Modify the input prompt to better guide the model in using the context; clearer instructions may help the model understand how to incorporate the context into its response
If the model consistently fails to use context, it may require retraining or fine-tuning with more examples that emphasize the importance of context utilization

Was this page helpful?

Questions & Discussion