PromptWizard

When to use PromptWizard

PromptWizard suits open-ended tasks where the right framing for a prompt is not obvious upfront. It works by mutating a prompt into several different framings, then critiquing and refining the best of those framings over a set number of iterations, rather than reacting to specific failures the way ProTeGi does.

PromptWizard runs from the SDK below, or from the platform’s Run Optimization drawer; see Run an optimization for the UI walkthrough.

Parameters

Parameter	On-screen label	Default	Description
teacher_generator	-	required, no default	Generator used for critique and refinement. Its prompt_template must be the passthrough "{prompt}"; PromptWizard fills it with its own critique-and-refine prompts, not your task prompt. Candidate prompts are run against your dataset by a generator PromptWizard manages internally, not by teacher_generator and not by a second generator you supply
mutate_rounds	Mutated Rounds	3	Number of mutation rounds used to generate prompt variations
refine_iterations	Refined Iterations	2	Number of full mutate, score, and refine cycles run on the best candidates; raising it repeats the mutation rounds again each cycle, not just the refine step
beam_size	Beam size	2 prefilled in the UI, 1 in the SDK	Number of top-scoring prompts carried forward at each round

This table covers only PromptWizard’s own tuning knobs. evaluator, data_mapper, dataset, and initial_prompts (required on every optimize() call) are shared by every optimizer’s optimize() call and are covered in the SDK reference.

Raising mutate_rounds, refine_iterations, or beam_size makes PromptWizard explore or refine more before it settles, at the cost of more generator and evaluator calls per run.

Usage

Before running the example below:

• Install: pip install agent-opt
• FI keys: get fi_api_key and fi_secret_key from Admin Settings, or set FI_API_KEY/FI_SECRET_KEY as environment variables and drop them from the Evaluator call below
• Model key: export OPENAI_API_KEY as an environment variable, since the example below passes gpt-4o-mini to LiteLLMGenerator; there’s no field in the code to put it in

The dataset below is a small inline list of dicts; see Optimize from the SDK for loading your own data instead.

from fi.opt.optimizers import PromptWizardOptimizer
from fi.opt.generators import LiteLLMGenerator
from fi.opt.datamappers import BasicDataMapper
from fi.opt.base.evaluator import Evaluator

# Dataset: a list of dicts, one per example. Keys must cover whatever the
# prompt template and key_map below reference, here just "article".
my_dataset = [
    {"article": "The James Webb Space Telescope has captured its clearest images yet of a distant exoplanet's atmosphere, revealing traces of carbon dioxide and methane."},
    {"article": "Researchers have discovered a new enzyme that breaks down PET plastic at room temperature, far faster than any previously known enzyme."},
]

# Teacher model used for critique and refinement; see teacher_generator in
# the table above. Your task prompt is passed to initial_prompts on
# optimize() below.
generator = LiteLLMGenerator(
    model="gpt-4o-mini",
    prompt_template="{prompt}"
)

# Evaluator that scores each candidate prompt.
# eval_template: see the built-in templates linked below
# eval_model_name: see the evaluator models linked below
evaluator = Evaluator(
    eval_template="summary_quality",
    eval_model_name="turing_flash",
    fi_api_key="your_key",
    fi_secret_key="your_secret"
)

# Maps generator output and dataset fields to what the evaluator expects.
# "article" must match a key in my_dataset; "generated_output" is filled
# in automatically by the optimizer, not a dataset field.
data_mapper = BasicDataMapper(
    key_map={"input": "article", "output": "generated_output"}
)

# mutate_rounds, refine_iterations, and beam_size here match the defaults
# in the table above and can be omitted; shown so they're easy to change.
optimizer = PromptWizardOptimizer(
    teacher_generator=generator,
    mutate_rounds=3,
    refine_iterations=2,
    beam_size=1
)

result = optimizer.optimize(
    evaluator=evaluator,
    data_mapper=data_mapper,
    dataset=my_dataset,
    # initial_prompts holds the starting prompt PromptWizard mutates and refines
    initial_prompts=["Summarize this article: {article}"]
)

# Read the optimized prompt and its score off the result
print(f"Final score: {result.final_score:.4f}")
print(f"Best prompt:\n{result.best_generator.get_prompt_template()}")

eval_template accepts any of the built-in evaluation templates; eval_model_name accepts any of the evaluator models.

A successful run prints something like:

Final score: 0.8214
Best prompt:
Summarize this article in 2-3 sentences, covering the main finding and its significance.

The exact score and wording vary by run. If a key_map value doesn’t match a field in the dataset (here, article), nothing raises or names the mismatch: the field is silently missing from what the evaluator sees. Make sure every value in key_map matches a key present in your dataset’s dicts.

result holds more than final_score and best_generator; see OptimizationResult for the full field list, and Read the result for how to take best_generator’s prompt into production.

Keep exploring