Image Evaluation

import os
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
    fi_base_url="https://api.futureagi.com",
)

print("Evaluator client initialized")

You should see Evaluator client initialized printed with no errors. A bad key raises InvalidAuthError at the first evaluate() call, not here.

ImageInstructionAdherence scores how well a generated image follows the prompt that produced it. Here’s one row of a T2I (text-to-image) dataset: a prompt, the image generated from it, and the category the prompt targets. In production this comes from your generation pipeline’s output log, not a hand-typed dict.

from fi.evals.templates import ImageInstructionAdherence

datapoint = {
    "prompt": "a pair of white sneakers with a wavy sole design, product photo on a plain background",
    "image_url": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
    "category": "product-fidelity",
}

alignment_template = ImageInstructionAdherence()

alignment_result = evaluator.evaluate(
    eval_templates=[alignment_template],
    inputs=[{
        "instruction": datapoint["prompt"],
        "images": [datapoint["image_url"]],
    }],
    model_name="turing_flash",
)

alignment = alignment_result.eval_results[0]
print(alignment.output)
print(alignment.reason)

Example output (illustrative, depends on the actual image):

0.92
The image accurately shows a pair of white sneakers with a wavy sole design against a plain background.

output carries the adherence score for this eval (0 to 1, higher is better), reason explains it in plain language. model_name picks the evaluator model that scores the image; turing_flash is a fast default.

SyntheticImageEvaluator scores how confident the model is that an image was AI-generated rather than captured by a camera. Run it after the adherence check to confirm a generated asset is correctly recognized as synthetic, the signal a moderation or disclosure workflow needs before the asset ships.

from fi.evals.templates import SyntheticImageEvaluator

provenance_template = SyntheticImageEvaluator()

provenance_result = evaluator.evaluate(
    eval_templates=[provenance_template],
    inputs=[{
        "image": datapoint["image_url"],
    }],
    model_name="turing_flash",
)

provenance = provenance_result.eval_results[0]
print(provenance.output)
print(provenance.reason)

Example output (illustrative, depends on the actual image):

0.88
The smooth textures and uniform lighting are consistent with AI-generated product photography rather than a real camera capture.

output carries the confidence that the image is AI-generated (0 to 1, higher means more confident it’s synthetic), reason explains the visual cues behind the score. A low score here means the model reads the image as camera-captured, worth a second look if your pipeline expects every asset in this dataset to be generated.

Loop the alignment check over every datapoint and collect the pass rate. This is where a real dataset load belongs: one small list here, or a pd.read_csv(...) / json.load(...) over your own generation log in production.

datapoints = [
    {
        "prompt": "a pair of white sneakers with a wavy sole design, product photo on a plain background",
        "image_url": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "category": "product-fidelity",
    },
    {
        "prompt": "a red leather handbag with gold buckles on a wooden table",
        "image_url": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "category": "product-mismatch",
    },
    {
        "prompt": "a pair of running shoes shown mid-stride on a track",
        "image_url": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "category": "context-mismatch",
    },
]

PASS_THRESHOLD = 0.5  # matches the SDK's own passed = score >= 0.5

results = []

for dp in datapoints:
    result = evaluator.evaluate(
        eval_templates=[alignment_template],
        inputs=[{
            "instruction": dp["prompt"],
            "images": [dp["image_url"]],
        }],
        model_name="turing_flash",
    )
    verdict = result.eval_results[0]
    results.append({
        "category": dp["category"],
        "score": verdict.output,
        "passed": verdict.output >= PASS_THRESHOLD,
        "reason": verdict.reason,
    })

pass_rate = sum(1 for r in results if r["passed"]) / len(results)
print(f"Pass rate: {pass_rate:.0%} across {len(results)} images")

You should see a pass rate printed. The three rows here reuse one image against different prompts on purpose, so the mismatched rows score low. Group results by category on your own dataset to spot which prompt categories the generator handles worst.

When passed is False, reason tells you why the image didn’t match.

failures = [r for r in results if not r["passed"]]

for f in failures:
    print(f["category"], f["score"])
    print(f["reason"])

Example output (illustrative, from the batch above):

product-mismatch 0.05
The image shows sneakers, not a handbag, so the prompt's subject is not represented.
context-mismatch 0.30
The image shows sneakers in a static product shot, not mid-stride on a track.

Reading the reasons above, both failures trace back to the prompt describing a subject or setting the source image never had, not a scoring bug. The fix is upstream: tighten the generation prompt (or point it at a matching image) and rerun the same row through Step 2 to confirm the score crosses PASS_THRESHOLD.