Understanding optimization

Optimization finds a better-scoring prompt without hand-tuning

Optimization takes a prompt you already have and the evals you use to score it, then searches for a version of that prompt that scores higher, without you hand-tuning the wording yourself. Most of what follows describes that process against dataset rows.

A run fixes its setup before anything runs

An optimization run fixes four things at the start and doesn’t change them while it’s going:

• A prompt column to improve
• An optimizer algorithm that generates candidate prompts
• A model that produces outputs
• The evals you select as its objective

From that fixed setup, a run produces trials. One is the baseline trial, and it holds your original prompt exactly as it stood before the run started. Every trial after it is a numbered variation trial, holding one new candidate prompt the optimizer decided to try next. How many variation trials a run produces is set when the run is configured; see Run an optimization.

A trial is one candidate prompt, scored row by row

Every trial, baseline or variation, holds one candidate prompt and one result per dataset row. The baseline trial runs your original prompt against those rows, each variation trial its own candidate. That row-by-row result is where the actual measurement happens, not the prompt itself.

Each row result carries a score and a reason for every eval you selected. Select three evals and a single row leaves three scores and three reasons behind it, one pair per eval.

From there, each trial’s row scores roll up into a single ranking:

• The mean of a trial’s row scores becomes that trial’s average score
• The average score ranks the trial against the other variation trials, not the baseline
• The run reports the highest variation average score as the best score. The baseline’s average score isn’t a candidate in that ranking, it’s the comparison line the variations are measured against, so a run can finish with a best score below the baseline

A finished run keeps its winning candidate prompt alongside the baseline, so it’s right there to review; see Read optimization results for how to walk through it.

Here’s how everything above fits together:

flowchart TD
accTitle: What an optimization run is made of
accDescr: An optimization run fixes a prompt column, an optimizer algorithm, and a model, and takes a set of evals as its objective. It produces a baseline trial and variation trials, each holding a row result per dataset row. Each row result carries a score and a reason per eval, and the mean of those becomes the trial's average score. Variation trials are ranked by that average score into a best score, which is measured against the baseline trial's own average score rather than ranked alongside it.
Run["Optimization run"] -->|"fixes"| Column["Prompt column"]
Run -->|"fixes"| Optimizer["Optimizer algorithm"]
Run -->|"fixes"| Model["Model"]
Run -->|"objective"| Evals["Selected evals"]
Run -->|"produces"| Baseline["Baseline trial"]
Run -->|"produces"| Variation["Variation trial"]
Baseline -->|"holds"| Row["Row result"]
Variation -->|"holds"| Row
Row -->|"per eval"| ScoreReason["Score and reason"]
ScoreReason -->|"mean"| Avg["Trial average score"]
Avg -->|"ranks (variations only)"| Best["Best score"]
Baseline -->|"average score"| BaselineScore["Baseline score"]
Best -->|"measured against"| BaselineScore

What you watch while it runs

A run moves through four steps in order: onboarding (initializing the run), running the baseline eval, starting trials, and finalizing the optimization. Its status is a coarser read on that same progress: Queue means the run is waiting to start, before onboarding begins. Running covers all four steps, from onboarding through finalizing. Completed means finalizing has finished. A run can also end Failed or Cancelled instead of Completed.

You don’t wait for Completed to see anything. Each trial becomes readable as soon as it finishes scoring, one at a time as the run works through them, rather than all arriving together at the end.

A run scores at most 50 rows, and evals decide what counts as better

A run scores at most 50 dataset rows. Because the evals you pick are the objective, they’re the entire definition of “better” for that run: change which evals are attached and the very same set of candidate prompts can rank in a different order.

Two surfaces, one engine: datasets and Simulation

The same engine that optimizes a prompt column against dataset rows also optimizes an agent’s prompt from a Simulation run. The optimizer algorithms, the trial structure, and the scoring shape described above carry over; the rest of the setup differs. The agent surface has no prompt column, since its run hangs off a test execution instead. Its status set drops Cancelled, and its fourth step is named Finalizing agent prompt rather than Finalizing optimization. The sample changes too: instead of dataset rows capped at 50, a Simulation run samples 5 to 10 scenarios from the test execution, or all of them when the execution has 10 or fewer.

Keep exploring