End-to-End Prompt Optimization

Take a vague baseline prompt, configure a full agent-opt pipeline, and run RandomSearchOptimizer to rank prompt variants by score.

📝

TL;DR

Wire up every piece of an agent-opt optimization run: a dataset, a generator bound to a baseline prompt, an evaluator, a data mapper, and RandomSearchOptimizer. Score the vague baseline first, then run the optimizer and compare.

Time	Difficulty	Package
20 min	Intermediate	`agent-opt`

Prerequisites

Future AGI account → app.futureagi.com
API keys: FI_API_KEY and FI_SECRET_KEY (see Get your API keys)
An OpenAI API key (the generator’s gpt-4o-mini and the optimizer’s gpt-4o teacher model both call OpenAI through LiteLLM)
Python 3.11

Install

pip install agent-opt

export FI_API_KEY="your-api-key"
export FI_SECRET_KEY="your-secret-key"
export OPENAI_API_KEY="your-openai-api-key"

Note

RandomSearchOptimizer bills your OpenAI key beyond the generator’s own calls: each optimize() run adds num_variations=5 gpt-4o teacher calls plus a gpt-4o-mini generation per dataset row per variation, on top of the Future AGI evaluator calls.

Tutorial

Prepare the dataset and baseline prompt

The dataset is the set of inputs the optimizer scores every prompt variant against. The baseline prompt is deliberately vague, so there’s a real gap for optimization to close.

dataset = [
    {
        "article": "The James Webb Space Telescope captured detailed images of the Pillars of Creation.",
    },
    {
        "article": "Researchers discovered an enzyme that rapidly breaks down plastic.",
    },
]

# Deliberately vague: no length constraint, no instruction on what to keep
baseline_prompt = "Summarize this: {article}"

Wire the generator, evaluator, and data mapper

The generator binds the baseline prompt to a model configuration; the optimizer calls it to produce output for every candidate prompt it tries. The evaluator scores each candidate’s output, it’s the objective function the optimizer optimizes against. The data mapper connects dataset fields to what the evaluator expects.

from fi.opt.generators import LiteLLMGenerator
from fi.opt.base.evaluator import Evaluator
from fi.opt.datamappers import BasicDataMapper

generator = LiteLLMGenerator(
    model="gpt-4o-mini",
    prompt_template=baseline_prompt,
)

evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
)

data_mapper = BasicDataMapper(
    key_map={
        "input": "article",
        "output": "generated_output",
    }
)

generated_output is a sentinel value BasicDataMapper recognises and replaces with each variant’s output. Keep it exactly as shown.

Note

summary_quality is one of Future AGI’s built-in evals. Swap it for any other built-in template that matches your task.

Score the vague baseline now, before touching the optimizer, so you have a real number to compare against later:

baseline_outputs = [generator.generate(row) for row in dataset]
mapped = [
    data_mapper.map(output, row)
    for output, row in zip(baseline_outputs, dataset)
]
results = evaluator.evaluate(mapped)
baseline_score = sum(r.score for r in results) / len(results)

print(f"Baseline score: {baseline_score:.3f}")

You should see a Baseline score between 0 and 1. It’s the vague prompt’s true score, expect it to be middling, illustrating exactly the gap optimize() is meant to close.

Select the optimizer

RandomSearchOptimizer generates prompt variations with a teacher model and scores each with the evaluator. Future AGI ships 5 other strategies behind the same optimize() call:

Bayesian Search for few-shot example selection
ProTeGi for targeted edits
Meta-Prompt for higher-level rewrites
GEPA for evolutionary search
PromptWizard for multi-stage refinement

Swapping the optimizer class doesn’t change the rest of the workflow, the dataset, evaluator, and data mapper stay the same.

from fi.opt.optimizers import RandomSearchOptimizer

optimizer = RandomSearchOptimizer(
    generator=generator,
    teacher_model="gpt-4o",
    num_variations=5,
)

You should see an optimizer instance bound to generator, ready to call optimize(); nothing runs until the next step.

Run the optimization and inspect the results

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
)

print(f"Baseline score: {baseline_score:.3f}")
print(f"Final score: {result.final_score:.3f}")
print(f"Delta: {result.final_score - baseline_score:+.3f}")
print(f"Best prompt:\n{result.best_generator.get_prompt_template()}")

for i, iteration in enumerate(result.history):
    print(f"Round {i + 1}: score={iteration.average_score:.3f}")

You should see the final score beat the baseline score from step 2, a positive delta, the text of the highest-scoring prompt variant, and one score per round as Random Search works through the 5 variations it generated.

optimize() handled evaluation, the search loop, and ranking. What’s left is a scored, winning prompt that measurably beats the vague baseline.

Troubleshooting

Symptom	Cause	Fix
`ValueError` naming `fi_api_key` / `fi_secret_key` when constructing `Evaluator(...)`	`FI_API_KEY` / `FI_SECRET_KEY` weren’t exported before the script started	Export both keys in the same shell session, then rerun
`ValueError: Invalid configuration...` constructing `Evaluator()`	Neither a `metric` nor the `eval_template` + `eval_model_name` pair was passed	Pass `eval_template` and `eval_model_name` together, or a local `metric` instead
LiteLLM raises an OpenAI auth error	`OPENAI_API_KEY` missing; the generator’s `gpt-4o-mini` and the optimizer’s `gpt-4o` teacher model both route through LiteLLM to OpenAI	Export `OPENAI_API_KEY` alongside the FI keys
The evaluator scores an empty or missing `output` input and returns a flat/zero score	`BasicDataMapper`’s `key_map` output value isn’t the literal `generated_output` sentinel, so the mapper silently drops the field	Keep `"output": "generated_output"` exactly as shown
`result.final_score` barely moves after optimization	`num_variations` too low, or `eval_template` doesn’t match what the dataset actually tests	Raise `num_variations`, and match `eval_template` to the task
`optimizer.optimize()` takes several minutes	Expected for `RandomSearchOptimizer` with `num_variations=5`: each variation calls the `gpt-4o` teacher model plus the evaluator	Lower `num_variations` for a faster first run, or wait it out

Once you have a winning prompt, save it as a versioned template you can serve without a redeploy. See Prompt Versioning.

Was this page helpful?

Questions & Discussion