Prompt Optimization: Improve a Prompt Automatically

The optimizer needs labeled examples: inputs where you know what a good output looks like. This is your ground truth for scoring.

# Your test dataset: short multi-fact articles that need precise extraction
dataset = [
    {
        "article": "A Phase III trial across 47 hospitals in 12 countries found that combining pembrolizumab with an mRNA vaccine reduced melanoma recurrence by 44% versus pembrolizumab alone over 3 years, though grade 3+ adverse events rose from 11% to 18%.",
        "target_summary": "A 12-country Phase III trial found pembrolizumab plus an mRNA vaccine cut melanoma recurrence by 44% over 3 years, though grade 3+ adverse events rose from 11% to 18%.",
    },
    {
        "article": "The European Central Bank raised interest rates by 25 basis points to 4.5%, the tenth consecutive hike since July 2022. Core eurozone inflation fell to 4.3% from 5.3%, but remains above the 2% target. Markets now price in rate cuts starting Q2 2024.",
        "target_summary": "The ECB raised rates 25bp to 4.5% (tenth straight hike) as core eurozone inflation fell to 4.3%, still above the 2% target; markets expect cuts from Q2 2024.",
    },
    {
        "article": "Meta's Llama 3, trained on 15 trillion tokens across 16,384 H100 GPUs, scores 82.0 on MMLU, approaching GPT-4 on several benchmarks. It lags in math reasoning (48.2 vs GPT-4's 67.1 on MATH) and is licensed for commercial use under 700M monthly active users.",
        "target_summary": "Meta's Llama 3 (70B, 15T tokens) scores 82.0 on MMLU near GPT-4 level but lags in math (48.2 vs 67.1 on MATH); commercially licensed for companies under 700M MAU.",
    },
    {
        "article": "Japan's population fell by 837,000 in 2023, the largest drop since records began in 1968, with fertility at 1.20. PM Kishida announced a $25 billion child-rearing package including childcare subsidies and 80%-salary parental leave for up to 28 weeks.",
        "target_summary": "Japan lost 837K people in 2023 (record drop) with fertility at 1.20; Kishida's $25B support package includes childcare subsidies and 80%-salary parental leave.",
    },
]

# Deliberately bad baseline prompt: vague, no structure, no constraints
baseline_prompt = "Tell me about this: {article}"

The Evaluator scores each candidate prompt’s outputs during optimization. It uses Future AGI’s Turing models to judge output quality against the source article, for example how well the summary captures key information.

from fi.opt.base import Evaluator

evaluator = Evaluator(
    eval_template="summary_quality",  # Turing model, scores how well the summary captures the article
    eval_model_name="turing_flash",   # fast evaluator model, keeps each optimization round short
)

Before optimizing, measure the baseline so you have a comparison point.

from openai import OpenAI
from fi.opt.datamappers import BasicDataMapper

client = OpenAI()

# "generated_output" is the literal key the optimizer fills in during optimize():
# using the same mapper here keeps baseline and optimized scores on one code path
data_mapper = BasicDataMapper(
    key_map={
        "input":  "article",
        "output": "generated_output",
    }
)

baseline_scores = []
for item in dataset:
    prompt = baseline_prompt.format(article=item["article"])
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    example = {**item, "generated_output": response.choices[0].message.content}

    baseline_result = evaluator.evaluate(data_mapper.map(example))
    baseline_scores.append(baseline_result[0].score)

baseline_avg = sum(baseline_scores) / len(baseline_scores)
print(f"Baseline average score: {baseline_avg:.3f}")

You should see a low score, well under 0.5. The vague baseline prompt gives the model no structure to follow, so summaries drift and miss key facts.

MetaPromptOptimizer uses a teacher model (GPT-4o) to iteratively rewrite and improve the prompt. Each round generates candidate prompts, scores them with the evaluator, and keeps the best.

from fi.opt.generators import LiteLLMGenerator
from fi.opt.optimizers import MetaPromptOptimizer

# Teacher model: the LLM that rewrites prompts
teacher = LiteLLMGenerator(model="gpt-4o", prompt_template="{prompt}")

optimizer = MetaPromptOptimizer(
    teacher_generator=teacher,
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
    initial_prompts=[baseline_prompt],
    task_description="Generate a concise, one-sentence news summary that captures the key fact and impact. Keep the {article} placeholder exactly as written.",
    eval_subset_size=4,  # evaluate all 4 examples per round
)

This takes 2-5 minutes depending on dataset size and number of rounds. You should see the optimizer print its round-by-round progress as it rewrites and rescopes the prompt.

Print the before/after scores and pull the winning prompt off the result object.

print(f"\n--- Optimization Results ---")
print(f"Baseline score:  {baseline_avg:.3f}")
print(f"Optimized score: {result.final_score:.3f}")
print(f"Improvement:     +{result.final_score - baseline_avg:.3f}\n")

print("Best prompt found:")
print("-" * 60)
best_prompt = result.best_generator.get_prompt_template()
print(best_prompt)
print("-" * 60)

# Show round-by-round progress
print("\nOptimization history:")
for i, iteration in enumerate(result.history):
    print(f"  Round {i+1}: score={iteration.average_score:.3f}")

Illustrative output on this dataset:

--- Optimization Results ---
Baseline score:  0.421
Optimized score: 0.847
Improvement:     +0.426

Best prompt found:
------------------------------------------------------------
Write a single, precise sentence that summarizes the most
important finding or event in the article, including any
key statistic, named entity, or deadline. Focus on what
is new, not background information.

Article: {article}
------------------------------------------------------------

Optimization history:
  Round 1: score=0.531
  Round 2: score=0.673
  Round 3: score=0.741
  Round 4: score=0.804
  Round 5: score=0.847

The score climbs round over round as the teacher model rewrites the prompt against the evaluator’s feedback.

Slot the winning template into your own call path and run it on an unseen article.

from openai import OpenAI

client = OpenAI()

def summarize(article: str) -> str:
    # Slot the winning prompt template
    assert "{article}" in best_prompt, "optimized prompt is missing the {article} placeholder"
    prompt = best_prompt.replace("{article}", article)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content


# Test it on a new article
test_article = """
NASA's Artemis III mission has been delayed until 2027 due to spacesuit development
challenges. The mission was originally planned for 2025 and would be the first
crewed lunar landing since Apollo 17 in 1972.
"""

print(summarize(test_article))
# → "NASA's Artemis III lunar landing has been postponed to 2027 due to spacesuit delays."

You should see a single, fact-dense sentence, not the rambling output the baseline prompt produced in Step 3.