Caption Hallucination
Detects hallucinated or fabricated details in image captions that aren't grounded in the visual input.
Caption Hallucination checks whether a caption describes only what’s actually visible in an image, or invents details the image doesn’t support. Run it to catch fabricated captions before they ship.
What it does
Caption Hallucination is an LLM-as-Judge eval. It reads the image and its caption, then flags whether the caption introduces details not grounded in the visual input.
Input
| Required Input | Type | Description |
|---|---|---|
image | string | URL or file path to the image being captioned |
caption | string | The caption text to evaluate |
Output
| Field | Type | Description |
|---|---|---|
| Result | Pass / Fail | Pass means the caption accurately represents what’s in the image without hallucination, Fail means the caption contains hallucinated elements |
| Reason | string | A plain-language explanation of the evaluation |
Run it from code
Call evaluate() with the template name and the eval’s required inputs. It returns the score and the reason.
Note
Before running: install the SDK and set FI_API_KEY / FI_SECRET_KEY. The model argument in the snippets is the evaluator model Future AGI uses to run the eval; turing_flash is a fast default.
from fi.evals import evaluate
result = evaluate(
"caption_hallucination",
image="https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp",
caption="old man",
model="turing_flash",
)
print(result.score)
print(result.reason)import { evaluate } from "@future-agi/ai-evaluation";
const result = await evaluate(
"caption_hallucination",
{
image: "https://www.esparklearning.com/app/uploads/2024/04/Albert-Einstein-generated-by-AI-1024x683.webp",
caption: "old man"
},
{ modelName: "turing_flash" }
);
console.log(result);When to use
Run Caption Hallucination wherever a caption is generated for an image and needs to stay grounded in what the image actually shows.
- Image captioning pipelines, to catch invented identities, locations, or actions
- RAG and retrieval systems that surface generated captions alongside images
- Hallucination checks on any workflow that pairs an image with generated text
What to do when Caption Hallucination fails
If the caption is evaluated as containing hallucinations, stick strictly to describing what is visibly present in the image. Avoid assumptions about people’s identities (unless clearly labeled or universally recognizable), the location or setting, time periods, actions before or after the captured moment, emotions or thoughts of subjects, and objects that are partially obscured or ambiguous.
Use qualifying language like “appears to be” or “what looks like” when uncertain, and focus on concrete visual elements rather than interpretations. For generic descriptions, stay high-level and avoid specifics that aren’t clearly visible.
Questions & Discussion