Basic Prompt Optimization

The optimizer needs examples to score candidate prompts against. Each row is a plain dict.

dataset = [
    {
        "article": "The James Webb Space Telescope has captured stunning new images of the Pillars of Creation, revealing intricate details of gas and dust clouds where new stars are forming.",
    },
    {
        "article": "Researchers have discovered a new enzyme that can break down plastics at record speed, offering a potential solution to the global plastic pollution crisis.",
    },
]

You should see nothing yet, this just defines the list in memory. Two rows is enough to run the tutorial; use 5-10 for a result you’d trust. The summary_quality template used later only scores article against the generated summary, so there’s no target_summary column to add.

LiteLLMGenerator wraps a model and a prompt template. This is the prompt you’re about to improve.

from fi.opt.generators import LiteLLMGenerator

initial_prompt = "Summarize this: {article}"
initial_generator = LiteLLMGenerator(
    model="gpt-4o-mini",
    prompt_template=initial_prompt,
)

Nothing runs yet. initial_generator now holds the model and template the optimizer will vary.

The Evaluator scores each candidate prompt’s output using a Future AGI eval template.

from fi.opt.base.evaluator import Evaluator

evaluator = Evaluator(
    eval_template="summary_quality",  # built-in template for summarization
    eval_model_name="turing_flash",   # judge model
)

Nothing runs yet. evaluator now holds the template and judge model it will use to score every candidate prompt’s output.

BasicDataMapper tells the optimizer which dataset column is the input and which generated field is the output to score.

from fi.opt.datamappers import BasicDataMapper

data_mapper = BasicDataMapper(
    key_map={"input": "article", "output": "generated_output"}
)

"generated_output" isn’t a column in dataset, it’s a reserved sentinel string. When a key_map value equals that exact string, BasicDataMapper.map substitutes the text the generator produced for that example instead of looking up a dataset column. Any other value is looked up in the dataset row as-is.

The optimizer only reports the score of its best variation, so score the unmodified initial_prompt first to have a number worth comparing against.

baseline_outputs = [initial_generator.generate(example) for example in dataset]
baseline_inputs = [
    data_mapper.map(output, example)
    for output, example in zip(baseline_outputs, dataset)
]
baseline_results = evaluator.evaluate(baseline_inputs)
baseline_score = sum(r.score for r in baseline_results) / len(baseline_results)
print(f"Baseline score: {baseline_score:.4f}")

You should see a Baseline score between 0 and 1. Keep that number, you’ll compare result.final_score against it once the optimizer finishes.

RandomSearchOptimizer uses a teacher model to write prompt variations, then scores each one with the Evaluator.

from fi.opt.optimizers import RandomSearchOptimizer

optimizer = RandomSearchOptimizer(
    generator=initial_generator,
    teacher_model="gpt-4o",  # writes the prompt variations
    num_variations=5,        # how many variations to generate
)

Nothing runs yet. optimizer now holds the generator, teacher model, and variation count it will use once you call optimize().

Hand the evaluator, data mapper, and dataset to the optimizer. This is the step that actually calls the teacher and generator models.

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=dataset,
)

This calls the teacher model to generate 5 prompt variations, runs each against your dataset, and scores every output. Larger datasets and higher num_variations take longer to run.

Pull the winning prompt and score out of result, and print every variation alongside the baseline you scored earlier.

print(f"Baseline Score: {baseline_score:.4f}")
print(f"Final Score: {result.final_score:.4f}")
print(f"Best Prompt Found:\n{result.best_generator.get_prompt_template()}")

for i, iteration in enumerate(result.history):
    print(f"\n--- Variation {i+1} ---")
    print(f"Score: {iteration.average_score:.4f}")
    print(f"Prompt: {iteration.prompt}")

You should see the Baseline Score printed again next to Final Score, the winning prompt text, and one entry per variation in result.history with its own score. result.final_score is the best variation found, so compare it against baseline_score to see whether optimization actually helped. The exact numbers depend on your dataset and the models you called, so don’t expect to match anyone else’s run.