Chunk Attribution

Checks whether a model references the provided context chunks at all when generating a response, a pass/fail RAG usage check.

Chunk Attribution checks whether the model acknowledges and draws on the retrieved context chunks at all when generating a response. Run it to catch cases where retrieval succeeded but the model ignored what it retrieved.

What it does

Chunk Attribution is an LLM-as-Judge eval. It reads the context chunks and the generated output, then returns a pass/fail on whether the output shows the model used that context.

Input

Required Input	Type	Description
`context`	`string` or `list[string]`	The contextual information provided to the model
`output`	`string`	The response generated by the language model

Output

Field	Type	Description
Result	`Pass / Fail`	Passed indicates the model acknowledged the context, Failed indicates potential issues
Reason	`string`	A plain-language explanation of the evaluation

Run it from code

Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.

Note

Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.

from fi.evals import evaluate

result = evaluate(
    "chunk_attribution",
    output="Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
    context=[
        "Paris is the capital and largest city of France.",
        "France is a country in Western Europe.",
        "Paris is known for its art museums and fashion districts."
    ],
    model="turing_flash",
)

print(result.score)
print(result.reason)

import { evaluate } from "@future-agi/ai-evaluation";

const result = await evaluate(
  "chunk_attribution",
  {
    output: "Paris is the capital city of France. It is a major European city and a global center for art, fashion, and culture.",
    context: [
      "Paris is the capital and largest city of France.",
      "France is a country in Western Europe.",
      "Paris is known for its art museums and fashion districts."
    ]
  },
  { modelName: "turing_flash" }
);

console.log(result);

When to use

Run Chunk Attribution wherever you need a quick, binary check on whether retrieved context is being used at all.

RAG and retrieval pipelines, as a first-pass sanity check before measuring how well context is used
Debugging a generator that seems to ignore retrieved documents
Custom pipelines where you want a Pass/Fail signal rather than a graded score

What to do when Chunk Attribution fails

Ensure that the context provided is relevant and sufficiently detailed for the model to utilize effectively. Irrelevant context might be ignored
Modify the input prompt to explicitly guide the model to use the context, for example “Using the provided documents, answer…”
Check the retrieval mechanism: is the correct context being retrieved and passed to the generation model
If the model consistently fails to use context despite relevant information and clear prompts, it may require fine-tuning with examples that emphasize context utilization

Was this page helpful?

Questions & Discussion