Comparing Optimizers

A customer support task: the baseline prompt is deliberately vague so there’s room for every optimizer to improve on it.

from fi.opt.base import Evaluator
from fi.opt.datamappers import BasicDataMapper
from fi.opt.generators import LiteLLMGenerator

dataset = [
    {
        "question": "Can I switch from the annual plan to monthly, and do I get a refund?",
        "context": "Annual-to-monthly switches are done via Settings > Billing > Change Plan. "
                    "A prorated refund is issued for unused months minus a 10% early "
                    "termination fee. The switch takes effect immediately.",
        "ideal_response": "Switch via Settings > Billing > Change Plan. You'll get a "
                           "prorated refund for unused months minus a 10% early "
                           "termination fee, effective immediately.",
    },
    {
        "question": "Our SSO login is returning 403 errors after the last update.",
        "context": "SSO 403 errors after an update are usually a stale ACS URL or an "
                    "expired SAML certificate. Check Settings > Security > SSO, "
                    "re-download the SP metadata, and re-upload it to your IdP.",
        "ideal_response": "Check Settings > Security > SSO for a stale ACS URL or an "
                           "expired SAML certificate, then re-download and re-upload "
                           "the SP metadata to your IdP.",
    },
    # Two more rows (GDPR deletion, API latency) are in the notebook's full
    # 4-row dataset: trimmed here to keep the page moving.
]

# Deliberately vague: no structure, no constraints, will miss key details
baseline_prompt = "Help with this: {question}\n\nInfo: {context}"

completeness_mapper = BasicDataMapper(
    key_map={
        "input": "question",
        "output": "generated_output",
    }
)

Every optimizer in this run is scored on completeness: does the response cover everything the question asked for. The evaluator only sees question and generated_output. context feeds the generator prompt so the model has the facts to answer with, but it isn’t part of what the eval scores. Swap in any other built-in eval template to compare optimizers on a different axis.

completeness_evaluator = Evaluator(
    eval_template="completeness",
    eval_model_name="turing_flash",
)

Before running any optimizer, score the vague baseline prompt as-is so every later score has something to be measured against.

baseline_generator = LiteLLMGenerator(
    model="gpt-4o-mini",
    prompt_template=baseline_prompt,
)

baseline_outputs = [baseline_generator.generate(example) for example in dataset]
baseline_inputs = [
    completeness_mapper.map(output, example)
    for output, example in zip(baseline_outputs, dataset)
]
baseline_results = completeness_evaluator.evaluate(baseline_inputs)
baseline_score = sum(r.score for r in baseline_results) / len(baseline_results)

print(f"Baseline score: {baseline_score:.3f}")

Expected output (illustrative):

Baseline score: 0.720

That’s the number every optimizer below needs to beat. A vague prompt with no structure and no constraints leaves the model guessing at what to include, so it’s a low bar on purpose.

RandomSearchOptimizer generates variations of the prompt with a teacher model and scores each one. It’s the fastest way to check whether an optimizer helps at all before spending a larger budget.

from fi.opt.optimizers import RandomSearchOptimizer

random_optimizer = RandomSearchOptimizer(
    generator=baseline_generator,
    teacher_model="gpt-4o",
    num_variations=5,
)

random_result = random_optimizer.optimize(
    evaluator=completeness_evaluator,
    data_mapper=completeness_mapper,
    dataset=dataset,
)

print(f"Random Search score: {random_result.final_score:.3f}")
print(f"Variations tried: {len(random_result.history)}")

Expected output (illustrative, your scores will vary by model and dataset):

Random Search score: 0.780
Variations tried: 5

That’s 0.720 → 0.780, a +0.06 jump over the baseline for five cheap variations: a good sign it’s worth spending more budget on the optimizers below.

BayesianSearchOptimizer keeps the instruction text fixed and searches for the best number and combination of few-shot examples to append to it.

from fi.opt.optimizers import BayesianSearchOptimizer

bayesian_optimizer = BayesianSearchOptimizer(
    min_examples=1,
    max_examples=3,
    n_trials=10,
)

bayesian_result = bayesian_optimizer.optimize(
    evaluator=completeness_evaluator,
    data_mapper=completeness_mapper,
    dataset=dataset,
    initial_prompts=[baseline_prompt],
)

print(f"Bayesian Search score: {bayesian_result.final_score:.3f}")
print(f"Trials run: {len(bayesian_result.history)}")

Expected output (illustrative):

Bayesian Search score: 0.834
Trials run: 10

That’s 0.720 → 0.834, a +0.114 jump over the baseline, better than Random Search, at the cost of running twice as many trials.

GEPAOptimizer breeds, mutates, and selects prompts over generations. It’s the most thorough (and most expensive) search here. max_metric_calls is a budget on .optimize(), not a constructor argument, so it stays out of GEPAOptimizer(...).

from fi.opt.optimizers import GEPAOptimizer

gepa_optimizer = GEPAOptimizer(
    reflection_model="gpt-4o",       # powerful model for reflection and mutation
    generator_model="gpt-4o-mini",   # model used by the prompts being optimized
)

# max_metric_calls is kept low here for a quick demo run. For production
# optimization, raise it to 80-200 evaluation calls.
gepa_result = gepa_optimizer.optimize(
    evaluator=completeness_evaluator,
    data_mapper=completeness_mapper,
    dataset=dataset,
    initial_prompts=[baseline_prompt],
    max_metric_calls=10,
)

print(f"GEPA score: {gepa_result.final_score:.3f}")
print(f"Candidates evaluated: {len(gepa_result.history)}")

Expected output (illustrative):

GEPA score: 0.861
Candidates evaluated: 8

That’s 0.720 → 0.861, the largest jump of the three (+0.141 over the baseline). GEPA’s evolutionary search spends its budget differently from the other two: history counts every candidate evaluation, not every generation, so this number tracks toward the max_metric_calls=10 budget above rather than a small generation count.

results = {
    "Random Search": random_result,
    "Bayesian Search": bayesian_result,
    "GEPA": gepa_result,
}

print(f"{'Optimizer':<18} {'Score':>8} {'Iterations':>12}")
print("-" * 40)
for name, result in results.items():
    print(f"{name:<18} {result.final_score:>8.3f} {len(result.history):>12}")

best_name = max(results, key=lambda k: results[k].final_score)
print(f"\nBest: {best_name}")
print(results[best_name].best_generator.get_prompt_template())

Expected output (illustrative, ranking depends on your task and models):

Optimizer             Score   Iterations
----------------------------------------
Random Search         0.780            5
Bayesian Search       0.834           10
GEPA                  0.861            8

Best: GEPA
You are a customer support agent. Answer using only the information in
the provided context. Cover every step, number, or link mentioned...