Comparing Prompt Optimizers

A customer support response task: the optimizer must improve how well the agent answers a question using the provided context. The baseline prompt is deliberately vague so each optimizer has room to improve it.

from fi.opt.base import Evaluator
from fi.opt.datamappers import BasicDataMapper
from fi.opt.generators import LiteLLMGenerator

# A few multi-constraint support scenarios: a vague prompt will miss key details.
# The full 8-example dataset used to build this cookbook is in the notebook linked above.
dataset = [
    {
        "question": "I signed up for the annual plan 3 months ago but now want to switch to monthly. Do I get a refund, and will I lose my team seats?",
        "context": "Annual-to-monthly switches happen via Settings → Billing → Change Plan. Unused months are refunded minus a 10% early termination fee. Seats are preserved, but the price rises from $8/month to $12/month, effective immediately.",
        "ideal_response": "Switch via Settings → Billing → Change Plan. You'll get a prorated refund minus a 10% fee, and keep your seats, though the price rises from $8/month to $12/month, effective immediately.",
    },
    {
        "question": "Our SSO integration broke after the latest update. Users get 403 errors through Okta, but direct login still works.",
        "context": "403 errors after an update are usually an expired SAML certificate or a changed ACS URL. Check Settings → Security → SSO for the ACS URL (it may now include /v2/), or regenerate the certificate under Settings → Security → Certificates. Changes propagate within 15 minutes.",
        "ideal_response": "This is likely a changed ACS URL or expired SAML certificate. Check Settings → Security → SSO for the ACS URL, or regenerate the certificate under Settings → Security → Certificates. Allow 15 minutes for changes to propagate.",
    },
    {
        "question": "We need to comply with GDPR. Can you delete all data for our EU users, and how do I prove it happened?",
        "context": "Submit deletion requests via Settings → Compliance → Data Deletion Request, by email domain or a CSV of user IDs. Deletion covers profiles, activity logs, and content, completing within 72 hours. A signed deletion certificate is emailed automatically as proof.",
        "ideal_response": "Submit a request via Settings → Compliance → Data Deletion Request, by email domain or CSV. Deletion completes within 72 hours and you'll receive a signed deletion certificate by email as proof.",
    },
]

# Deliberately vague baseline: no structure, no constraints, will miss key details
baseline_prompt = "Help with this: {question}\n\nInfo: {context}"

# context_adherence and chunk_utilization need context + output
context_mapper = BasicDataMapper(
    key_map={
        "output":  "generated_output",
        "context": "context",
    }
)

# completeness needs input + output
completeness_mapper = BasicDataMapper(
    key_map={
        "input":  "question",
        "output": "generated_output",
    }
)

Nothing runs yet: this block defines the dataset, baseline prompt, and mappers in memory. The dataset stays fixed across all three optimizers so the comparison is fair.

A good support response needs to be faithful to the docs (context adherence), use the relevant info (chunk utilization), and fully answer the question (completeness). Each optimizer below gets a different metric so you can compare how metric choice affects the winning prompt.

# Evaluator 1: context_adherence, does the response stick to the provided context?
context_adherence_evaluator = Evaluator(
    eval_template="context_adherence",
    eval_model_name="turing_flash",
)

# Evaluator 2: chunk_utilization, how effectively does the response use the context?
chunk_utilization_evaluator = Evaluator(
    eval_template="chunk_utilization",
    eval_model_name="turing_flash",
)

# Evaluator 3: completeness, does the response fully answer the question?
completeness_evaluator = Evaluator(
    eval_template="completeness",
    eval_model_name="turing_flash",
)

Any built-in eval template works here: the optimizer is metric-agnostic.

Before running any optimizer, measure how the vague baseline prompt scores on all three metrics so you have a comparison point.

import os
from openai import OpenAI
from fi.evals import Evaluator as FIEvaluator

client = OpenAI()
baseline_eval = FIEvaluator(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

def score_baseline(eval_template, build_inputs):
    scores = []
    for item in dataset[:2]:
        prompt = baseline_prompt.format(question=item["question"], context=item["context"])
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        output = response.choices[0].message.content
        result = baseline_eval.evaluate(
            eval_templates=eval_template,
            inputs=build_inputs(item, output),
            model_name="turing_flash",
        )
        scores.append(float(result.eval_results[0].output))
    return sum(scores) / len(scores)

baseline_context_adherence = score_baseline(
    "context_adherence", lambda item, output: {"output": output, "context": item["context"]}
)
baseline_chunk_utilization = score_baseline(
    "chunk_utilization", lambda item, output: {"output": output, "context": item["context"]}
)
baseline_completeness = score_baseline(
    "completeness", lambda item, output: {"output": output, "input": item["question"]}
)

print(f"Baseline context adherence: {baseline_context_adherence:.3f}")
print(f"Baseline chunk utilization: {baseline_chunk_utilization:.3f}")
print(f"Baseline completeness:      {baseline_completeness:.3f}")

Illustrative output:

Baseline context adherence: 0.437
Baseline chunk utilization: 0.402
Baseline completeness:      0.388

You should see three low scores. The vague baseline prompt gives the model no structure, so it drifts from the context and misses parts of the question. Each optimizer’s job below is to beat its corresponding baseline number.

ProTeGi generates localized edits to specific parts of the prompt, then tests each edit. It uses “textual gradients”: error-based feedback that guides targeted rewrites.

from fi.opt.optimizers import ProTeGi

teacher = LiteLLMGenerator(model="gpt-4o", prompt_template="{prompt}")

# Values kept low for a quick demo run (~5 min).
# For production optimization, increase: num_gradients=4, errors_per_gradient=4,
# beam_size=4, num_rounds=5, eval_subset_size=len(dataset)
protegi_optimizer = ProTeGi(
    teacher_generator=teacher,
    num_gradients=1,
    errors_per_gradient=1,
    prompts_per_gradient=1,
    beam_size=1,
)

print("Running ProTeGi with context adherence metric...")
protegi_result = protegi_optimizer.optimize(
    evaluator=context_adherence_evaluator,
    data_mapper=context_mapper,
    dataset=dataset,
    initial_prompts=[baseline_prompt],
    num_rounds=1,
    eval_subset_size=2,
)

print(f"ProTeGi score: {protegi_result.final_score:.3f}")
print(f"Rounds completed: {len(protegi_result.history)}")

Illustrative output:

Running ProTeGi with context adherence metric...
ProTeGi score: 0.892
Rounds completed: 1

You should see a score and a round count. The exact numbers depend on your dataset and models, so don’t expect to match this run.

GEPA uses an evolutionary approach: it breeds, mutates, and selects prompts over generations. It explores more diverse prompt styles than gradient-based methods like ProTeGi.

from fi.opt.optimizers import GEPAOptimizer

gepa_optimizer = GEPAOptimizer(
    reflection_model="gpt-4o",      # critiques failures and proposes mutations
    generator_model="gpt-4o-mini",  # the model whose prompt is being optimized
)

# max_metric_calls kept low for a quick demo. For a real run, use 80-200.
print("Running GEPA with chunk utilization metric...")
gepa_result = gepa_optimizer.optimize(
    evaluator=chunk_utilization_evaluator,
    data_mapper=context_mapper,
    dataset=dataset,
    initial_prompts=[baseline_prompt],
    max_metric_calls=8,
)

print(f"GEPA score: {gepa_result.final_score:.3f}")
print(f"Rounds completed: {len(gepa_result.history)}")

Illustrative output:

Running GEPA with chunk utilization metric...
GEPA score: 0.871
Rounds completed: 2

max_metric_calls counts every evaluation, including the calls spent scoring the initial prompt. Keep it well above your dataset size once you move past this demo.

PromptWizard runs a 3-stage pipeline: mutate (generate prompt variants), score (evaluate candidates), and critique-refine (improve the best candidate). It applies different thinking styles (analytical, creative, step-by-step) during mutation for diverse candidates.

from fi.opt.optimizers import PromptWizardOptimizer

# Values kept low for a quick demo run (~1 min).
# For production optimization, increase: mutate_rounds=3, refine_iterations=2,
# eval_subset_size=len(dataset)
promptwizard_optimizer = PromptWizardOptimizer(
    teacher_generator=teacher,
    mutate_rounds=1,
    refine_iterations=1,
    beam_size=1,
)

print("Running PromptWizard with completeness metric...")
pw_result = promptwizard_optimizer.optimize(
    evaluator=completeness_evaluator,
    data_mapper=completeness_mapper,
    dataset=dataset,
    initial_prompts=[baseline_prompt],
    task_description="Generate a helpful, context-grounded customer support response that addresses all parts of the question.",
    eval_subset_size=2,
)

print(f"PromptWizard score: {pw_result.final_score:.3f}")
print(f"Rounds completed: {len(pw_result.history)}")

Illustrative output:

Running PromptWizard with completeness metric...
PromptWizard score: 0.914
Rounds completed: 4

Collect the three results into one table, print each winning prompt, and show the round-by-round history for the top scorer.

results = {
    "ProTeGi (context adherence)": protegi_result,
    "GEPA (chunk utilization)": gepa_result,
    "PromptWizard (completeness)": pw_result,
}

baselines = {
    "ProTeGi (context adherence)": baseline_context_adherence,
    "GEPA (chunk utilization)": baseline_chunk_utilization,
    "PromptWizard (completeness)": baseline_completeness,
}

print("\n" + "=" * 66)
print(f"{'Strategy':<30} {'Baseline':>10} {'Score':>8} {'Delta':>8}")
print("=" * 66)

for name, result in results.items():
    baseline = baselines[name]
    delta = result.final_score - baseline
    print(f"{name:<30} {baseline:>10.3f} {result.final_score:>8.3f} {delta:>+8.3f}")

print("=" * 66)

# Show the winning prompt from each strategy
for name, result in results.items():
    prompt = result.best_generator.get_prompt_template()
    print(f"\n--- {name} ---")
    print(prompt[:200] + ("..." if len(prompt) > 200 else ""))

# Show round-by-round history for the best performer
best_name = max(results, key=lambda k: results[k].final_score)
best_result = results[best_name]
print(f"\n--- {best_name}: round history ---")
for i, iteration in enumerate(best_result.history):
    print(f"  Round {i+1}: score={iteration.average_score:.3f}")

Illustrative output:

==================================================================
Strategy                        Baseline    Score    Delta
==================================================================
ProTeGi (context adherence)        0.437    0.892   +0.455
GEPA (chunk utilization)           0.402    0.871   +0.469
PromptWizard (completeness)        0.388    0.914   +0.526
==================================================================

--- ProTeGi (context adherence) ---
You are a customer support agent. Answer the question using ONLY the information in the provided context. Be specific and include exact steps, numbers, or links where avail...

--- GEPA (chunk utilization) ---
As a friendly support agent, provide a clear, actionable answer to the customer's question. Use the context below as your knowledge base. Structure your response with the m...

--- PromptWizard (completeness) ---
You are an expert customer support agent. Your task is to answer the customer's question completely and accurately using the provided context. Include all relevant details: s...

--- PromptWizard (completeness): round history ---
  Round 1: score=0.731
  Round 2: score=0.812
  Round 3: score=0.867
  Round 4: score=0.914

You should see each strategy’s baseline score, optimized score, and the delta between them, plus the leading 200 characters of each winning prompt. “Works better” isn’t the point: the delta is. The numbers above are illustrative: your actual scores depend on the models and dataset subset used.