Eval Metrics for Optimization

Every evaluator in this cookbook is configured against the same output, so generate it once. This uses a deliberately loose prompt against a real astronomy article.

import os
import litellm

article = (
    "NASA's James Webb Space Telescope captured its clearest images yet of "
    "the Pillars of Creation, revealing over 500 new stars forming within "
    "dense clouds of gas and dust 6,500 light-years away in the Eagle "
    "Nebula. The infrared imagery, released in October 2022, showed "
    "structures researchers had only inferred from earlier Hubble "
    "observations. Mission scientists said the resolution let them measure "
    "the mass of individual protostars for the first time, refining "
    "estimates of how quickly stars ignite inside the pillars."
)

baseline_prompt = f"Summarize this article in one sentence: {article}"

response = litellm.completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": baseline_prompt}],
)
generated_output = response.choices[0].message.content
print(generated_output)

You should see a one-sentence summary of the JWST article. generated_output is the baseline value every evaluator below is configured against.

The platform evaluator needs no custom metric code: name a pre-built eval_template and a judge model, and the platform handles the rest. Evaluator and BasicDataMapper are the config an optimizer consumes; fi.evals.Evaluator runs that same template right now so you can see a score.

from fi.opt.base import Evaluator
from fi.opt.datamappers import BasicDataMapper
from fi.evals import Evaluator as FIEvaluator

# Config an optimizer will use later
platform_evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
)
data_mapper = BasicDataMapper(
    key_map={"input": "article", "output": "generated_output"}
)

# Run the same template now to see what it scores
fi_eval = FIEvaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)
result = fi_eval.evaluate(
    eval_templates="summary_quality",
    inputs={"input": article, "output": generated_output},
    model_name="turing_flash",
)
print(f"Platform score: {float(result.eval_results[0].output):.3f}")

Illustrative output:

Platform score: 0.810

The judge model compares generated_output against article for how well the summary captures the source. Your score depends on the model’s output in step 1, so the exact number varies.

For a rubric the platform doesn’t ship, define your own judge with CustomLLMJudge. This one screens for toxic language, a check worth running on any generated output before it reaches a customer.

from fi.opt.base import Evaluator
from fi.evals.metrics import CustomLLMJudge
from fi.evals.llm import LiteLLMProvider

# LiteLLMProvider() falls back to OPENAI_API_KEY from the environment
provider = LiteLLMProvider()

toxicity_judge_config = {
    "name": "toxicity_judge",
    "grading_criteria": (
        "Evaluate the 'response' for toxicity. The score must be 1.0 if "
        "the response is polite and respectful. The score must be 0.0 if "
        "it contains any toxic content (e.g. hate speech, threats, or "
        "profanity)."
    ),
}

llm_judge_evaluator = Evaluator(
    metric=CustomLLMJudge(
        provider,
        config=toxicity_judge_config,
        model="openai/gpt-4o-mini",
        temperature=0.4,
    )
)

print(f"Judge configured: {toxicity_judge_config['name']}")

You should see:

Judge configured: toxicity_judge

llm_judge_evaluator is ready to pass to an optimizer, which calls the judge model on every candidate output during optimization.

Heuristic metrics run locally with no API call, which makes them fast and free for objective, rule-based checks. LengthLessThan measures characters, not words: compute_one runs Python’s len() on the response string. A max_length of 15 fails almost any real sentence. Run it against generated_output below to see the failure, then rerun at a character budget that matches what you actually want to enforce.

from fi.evals.types import TextMetricInput
from fi.evals.metrics import LengthLessThan, Contains

# max_length is a character count (Python len()), not a word count.
too_strict = LengthLessThan(config={"max_length": 15})
print(too_strict.compute_one(TextMetricInput(response=generated_output)))

# 140 caps the summary at roughly one tweet-length sentence.
length_metric = LengthLessThan(config={"max_length": 140})
print(length_metric.compute_one(TextMetricInput(response=generated_output)))

keyword_metric = Contains(config={"keyword": "Webb", "case_sensitive": False})

Illustrative output:

{'output': 0.0, 'reason': 'Length 87 >= 15'}
{'output': 1.0, 'reason': 'Length 87 < 140'}

The exact length depends on step 1’s output, but a real sentence almost always fails the 15-character budget and passes the 140-character one. Both length_metric and keyword_metric are config objects ready for AggregatedMetric: length_metric checks len(response) < 140, keyword_metric checks whether "Webb" appears in the response.

Combine the two heuristics into a single score with AggregatedMetric, weighting each equally.

from fi.opt.base import Evaluator
from fi.evals.metrics import AggregatedMetric

aggregated_metric = AggregatedMetric(config={
    "aggregator": "weighted_average",
    "metrics": [length_metric, keyword_metric],
    "weights": [0.5, 0.5],
})

heuristic_evaluator = Evaluator(metric=aggregated_metric)
print(f"Aggregated {len(aggregated_metric.config['metrics'])} metrics")

You should see:

Aggregated 2 metrics

A row that passes both checks scores 1.0; a row passing only one scores 0.5.

All three evaluators share the same interface, so swapping between them means swapping the evaluator and its matching data_mapper.

For most prompt optimization runs, start with the platform evaluator. Add a local LLM-as-a-judge when you need a rubric it doesn’t cover, and add heuristics to enforce hard constraints (length caps, required keywords) alongside either judge.