Multimodal Evaluation: Images, Audio, and PDF

import os
from fi.evals import Evaluator

evaluator = Evaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

Check whether a caption accurately describes an image. Pass the image as a URL (or base64) and the caption as text.

# Accurate caption
result = evaluator.evaluate(
    eval_templates="caption_hallucination",
    inputs={
        "image": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "caption": "A pair of white sneakers with a wavy sole design.",
    },
    model_name="turing_small",
)

eval_result = result.eval_results[0]
print(f"Passed: {eval_result.output}")
print(f"Reason: {eval_result.reason}")

Try a hallucinated caption against the same image:

result = evaluator.evaluate(
    eval_templates="caption_hallucination",
    inputs={
        "image": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
        "caption": "A red leather handbag with gold buckles on a wooden table.",
    },
    model_name="turing_small",
)

eval_result = result.eval_results[0]
print(f"Passed: {eval_result.output}")
print(f"Reason: {eval_result.reason}")

Score whether an image was generated by AI or is a real photograph.

result = evaluator.evaluate(
    eval_templates="synthetic_image_evaluator",
    inputs={
        "image": "https://raw.githubusercontent.com/future-agi/cookbooks/main/ecom_agent/observe/generated_products/nike_air_max_sneakers.png",
    },
    model_name="turing_small",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

Get a Mean Opinion Score (MOS) assessment of audio quality. Pass the audio file as a URL or base64.

result = evaluator.evaluate(
    eval_templates="audio_quality",
    inputs={
        "input_audio": "https://storage.googleapis.com/cloud-samples-data/speech/brooklyn_bridge.flac",
    },
    model_name="turing_large",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

Check whether a TTS audio output accurately reflects the original text (including pronunciation, emphasis, and tone).

result = evaluator.evaluate(
    eval_templates="TTS_accuracy",
    inputs={
        "text": "Welcome to FutureAGI. Our platform helps you evaluate and optimize AI applications.",
        "generated_audio": "https://storage.googleapis.com/cloud-samples-data/speech/brooklyn_bridge.flac",
    },
    model_name="turing_large",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

Score how accurately OCR-extracted content matches the source PDF document.

result = evaluator.evaluate(
    eval_templates="ocr_evaluation",
    inputs={
        "input_pdf": "https://your-bucket.s3.amazonaws.com/sample-invoice.pdf",
        "json_content": '{"invoice_number": "INV-2024-001", "total": "$1,250.00", "date": "2024-03-15"}',
    },
    model_name="turing_large",
)

eval_result = result.eval_results[0]
print(f"Score:  {eval_result.output}")
print(f"Reason: {eval_result.reason}")

You can also run these evals directly from the FutureAGI platform without writing any code.

1. Go to Datasets and create or open a dataset
2. Add columns for your multimodal inputs (e.g. an image column with image URLs, or an audio column with audio URLs)
3. Click Add Evaluation and select a multimodal eval (e.g. caption_hallucination, audio_quality)
4. Map the eval’s required keys to your dataset columns (e.g. image → your image column, caption → your caption column)
5. Choose a Turing model and click Run
6. View scores alongside each row in the dataset

This is the same approach shown in the Dataset SDK cookbook and Dataset Management cookbook, but with multimodal columns instead of text-only.