Evaluate Images

Ensuring quality and alignment in AI-generated images is critical in text-to-image processes. A robust evaluation system verifies that the images adhere to the provided instructions, maintain logical consistency, and meet established quality standards. This evaluation process produces measurable insights into how closely the outputs match the input prompts, enabling continuous improvements in model accuracy and output reliability.

The process begins with structured inputs that include clear text instructions outlining the expected content and layout, as well as metadata that provides supplementary details, such as image URLs or contextual parameters. These inputs are processed through an evaluation pipeline where an evaluation agent applies predefined criteria to assess the generated images. The outcome is an alignment score, which is a numerical value that quantifies the degree to which the image meets the instructions and metadata driven insights that highlight strengths and pinpoint any deviations from the expected outcomes.

The following two evaluations focus on critical aspects of image quality, addressing both overall alignment with input prompts and the accurate representation of the provided instructions:

• Score Eval
• Eval Image Instruction

​

1. Score Eval

This evaluation measures the degree of linkage between textual instructions, input images, and output images. It ensures that the relationship between these elements is coherent and follows predefined rules or prompts. It is deal for validating general consistency across tasks where instructions, input images, and generated outputs must align logically. The output is a score that represents how well the input and outputs adhere to the specified rule.

Click here to read the eval definition of Score Eval

​

a. Using Future AGI Platform

Required Parameters

• Input: The text or instruction that serves as the reference for evaluation.
• Rule Prompt: A guideline or rule used to measure the linkage. This can include dynamic placeholders (e.g., {{column_name}}).

​

b. Using SDK

from fi.evals import EvalClient
from fi.evals.templates import ScoreEval
from fi.testcases import MLLMTestCase

test_case = MLLMTestCase(
    input="Score the sunset beach photo's composition and atmosphere",
    rule_string="""
    Evaluate:
    1. Golden hour lighting quality
    2. Rule of thirds composition
    3. Foreground to background balance
    4. Color harmony and mood
    """
)

template = ScoreEval(
    config={
        "input": {
            "type": "rule_string",
            "default": ["composition", "lighting", "atmosphere"]
        },
        "rulePrompt": {
            "type": "rule_prompt",
            "default": """
            Provide a score based on:
            - Composition (0-0.25): Rule of thirds, leading lines
            - Lighting (0-0.25): Quality of light, shadows, highlights
            - Atmosphere (0-0.25): Mood, emotional impact
            - Technical (0-0.25): Focus, exposure, clarity
            """
        }
    }
)

evaluator = EvalClient(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="https://api.futureagi.com"
)

result = evaluator.evaluate(eval_templates=[template], inputs=[test_case])

score = result.eval_results[0].metrics[0].value
reason = result.eval_results[0].reason

print(f"Evaluation Score: {score}")
print(f"Evaluation Reason: {reason}")

​

2. Eval Image Instruction

Scores the linkage between textual instructions and the resulting image based on specified criteria, whereas Score Eval examines overall coherence and adherence to instructions. Eval Image Instruction is ideal for cases where precise image representation is the main concern

Click here to read the eval definition of Eval Image Instruction

​

a. Using Future AGI Platform

Required Parameters

• Input: The instruction or textual description associated with the image (e.g., “A vibrant sunrise over a mountain”).
• Image URL: The URL of the image being evaluated.

Configuration Parameters

• Criteria: The evaluation standard that defines how the alignment is measured (e.g., colour accuracy, object representation, or stylistic features).

​

b. Using SDK

from fi.evals import EvalClient
from fi.evals.templates import ImageInstruction
from fi.testcases import MLLMTestCase

test_case = MLLMTestCase(
    input="A serene beach landscape photo taken from a wooden boardwalk",
    image_url="https://example.com/beach_photo.jpg"
)

template = ImageInstruction(
    config={
        "criteria": """
        Evaluate the image based on:
        1. Instruction clarity and specificity
        2. Image composition alignment
        3. Scene elements accuracy
        4. Overall visual quality
        """
    }
)

evaluator = EvalClient(
    fi_api_key="your_api_key",
    fi_secret_key="your_secret_key",
    fi_base_url="https://api.futureagi.com"
)

response = evaluator.evaluate(eval_templates=[template], inputs=[test_case])

score = response.eval_results[0].metrics[0].value
reason = response.eval_results[0].reason

print(f"Evaluation Score: {score}")
print(f"Evaluation Reason: {reason}")