Multimodal Evaluation: Images, Audio, and PDF

import os
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

You should see no output. evaluator is ready to score any of the templates below.

Check whether a caption accurately describes an image. Pass the image as a URL (or base64) and the caption as text.

result = evaluator.evaluate(
    eval_templates="caption_hallucination",
    inputs={
        "image": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "caption": "A pair of white sneakers with a wavy sole design.",
    },
    model_name="turing_small",
)

eval_result = result.eval_results[0]
print(f"Passed: {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You should see a passing verdict and a reason confirming the caption matches the sneaker image. Now try a caption that describes something else entirely:

result = evaluator.evaluate(
    eval_templates="caption_hallucination",
    inputs={
        "image": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "caption": "A red leather handbag with gold buckles on a wooden table.",
    },
    model_name="turing_small",
)

eval_result = result.eval_results[0]
print(f"Passed: {eval_result.output}")
print(f"Reason: {eval_result.reason}")

This time the verdict flips to failing. The reason explains what the caption claims that the image doesn’t show.

Score whether an image was generated by AI or is a real photograph.

result = evaluator.evaluate(
    eval_templates="synthetic_image_evaluator",
    inputs={
        "image": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
    },
    model_name="turing_small",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You should see a numeric score plus a reason describing the visual cues the model used.

Get a Mean Opinion Score (MOS) assessment of audio quality. Pass the audio file as a URL or base64.

result = evaluator.evaluate(
    eval_templates="audio_quality",
    inputs={
        "input_audio": "https://storage.googleapis.com/cloud-samples-data/speech/brooklyn_bridge.flac",
    },
    model_name="turing_large",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You should see a MOS-style score and a reason noting artifacts like noise or clipping, if any.

Check whether a TTS audio output accurately reflects the original text, including pronunciation, emphasis, and tone.

result = evaluator.evaluate(
    eval_templates="TTS_accuracy",
    inputs={
        "text": "Welcome to Future AGI. Our platform helps you evaluate and optimize AI applications.",
        "generated_audio": "https://storage.googleapis.com/cloud-samples-data/speech/brooklyn_bridge.flac",
    },
    model_name="turing_large",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You should see a score and a reason comparing the spoken audio against the source text.

Score how accurately OCR-extracted content matches the source PDF document. Substitute input_pdf with a publicly reachable URL to your own PDF, and json_content with the fields you expect the OCR pass to have extracted from it.

result = evaluator.evaluate(
    eval_templates="ocr_evaluation",
    inputs={
        "input_pdf": "https://your-public-url.example.com/your-document.pdf",
        "json_content": '{"invoice_number": "INV-2024-001", "total": "$1,250.00", "date": "2024-03-15"}',
    },
    model_name="turing_large",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You should see a score and a reason listing which fields matched or diverged from the PDF, based on the document and fields you supplied.

You can also run these evals directly from the Future AGI platform without writing any code.

1. Go to Datasets and create or open a dataset
2. Add columns for your multimodal inputs, for example an image column with image URLs, or an audio column with audio URLs
3. Click Add Evaluation and select a multimodal eval, for example caption_hallucination or audio_quality
4. Map the eval’s required keys to your dataset columns, for example image to your image column and caption to your caption column
5. Choose a Turing model and click Run
6. View scores alongside each row in the dataset

This is the same approach shown in the Dataset SDK cookbook, but with multimodal columns instead of text-only.