Skip to main content
Optimization is an approach of refining and improving prompts to achieve higher-quality, more consistent AI-generated responses. It is a key part of evaluation-driven development, allowing users to fine-tune their AI workflows based on structured evaluations rather than trial and error. Unlike experimentation, which compares different configurations of prompt, optimization focuses on iteratively improving a specific prompt using a feedback loop. By leveraging evaluations, scoring mechanisms, and iterative improvements, optimization ensures that prompts are more efficient, cost-effective, and aligned with business or application goals.

​
Why Optimization is Necessary?

Experimentation allows users to compare different prompt or model configurations, but it does not refine a single prompt in a systematic, data-driven way. Once an experiment identifies a well-performing prompt, optimization takes it a step further by making iterative improvements. This process enhances clarity, response quality, and efficiency while reducing ambiguity that can cause inconsistencies in AI outputs. Since LLMs generate responses probabilistically, even the same input can produce different outputs. Optimization ensures that prompts are structured to deliver the most consistent, high-quality results while minimising unnecessary token usage.

​
How Optimization Works?

An optimization task is initiated by defining its core components: a dataset of examples, an initial prompt to serve as a baseline, evaluation metrics to score performance, and an optimization algorithm to guide the process. These criteria define how improvements will be measured and ensure that changes lead to meaningful refinements.

​
Processing and Feedback Loop

The optimization process is managed by an Optimizer, which begins by running the initial prompt to establish a baseline performance score. The optimizer then enters an iterative loop: it programmatically modifies the prompt to create new candidates, runs them against the dataset to generate responses, and uses feedback from the evaluation metrics to guide the next round of changes. This iterative process continues across multiple cycles, with the optimizer intelligently exploring the prompt space to find the best-performing version.

​
Evaluation and Scoring

Throughout optimization, AI-generated responses are assessed using predefined evaluation metrics. These include:
  • Accuracy – How well does the response align with the expected outcome?
  • Fluency and Coherence – Is the response well-structured and natural?
  • Token Efficiency – Does the response avoid unnecessary word usage?
  • Relevance – Does the response directly address the given input?
Each iteration assigns a performance score to the prompt, and the optimizer uses these scores to track progress and identify improved versions.

​
Optimized Output Selection

Once the optimization is complete, the system compares the original prompt against the best-performing version found by the optimizer, highlighting measurable improvements. This optimized prompt is then ready for deployment.

​
Choosing an Optimization Strategy

The Prompt Optimizer library provides six different optimization algorithms, each with unique strengths and approaches to improving prompts. This guide helps you understand what each optimizer does and when to use it.

​
Algorithm Comparison

Bayesian Search

Smart few-shot optimization

Meta-Prompt

Deep reasoning refinement

ProTeGi

Error-driven improvement

PromptWizard

Creative exploration

GEPA

Evolutionary optimization

Random Search

Quick baseline testing

​
Quick Selection Guide

Use CaseRecommended OptimizerWhy
Few-shot learning tasksBayesian SearchIntelligently selects and formats examples
Complex reasoning tasksMeta-PromptDeep analysis of failures and systematic refinement
Improving existing promptsProTeGiFocused on identifying and fixing specific errors
Creative/open-ended tasksPromptWizardExplores diverse prompt variations
Production deploymentsGEPARobust evolutionary search with efficient budgeting
Quick experimentationRandom SearchFast baseline for comparison

​
Performance Comparison

OptimizerSpeedQualityCostBest Dataset Size
Bayesian Search⚡⚡⭐⭐⭐⭐💰💰15-50 examples
Meta-Prompt⚡⚡⭐⭐⭐⭐💰💰💰20-40 examples
ProTeGi⚡⭐⭐⭐⭐💰💰💰20-50 examples
PromptWizard⚡⭐⭐⭐⭐💰💰💰15-40 examples
GEPA⚡⭐⭐⭐⭐⭐💰💰💰💰30-100 examples
Random Search⚡⚡⚡⭐⭐💰10-30 examples
Speed: ⚡ = Slow, ⚡⚡ = Medium, ⚡⚡⚡ = Fast
Quality: ⭐ = Basic, ⭐⭐⭐⭐⭐ = Excellent
Cost: 💰 = Low, 💰💰💰💰 = High (based on API calls)

​
Detailed Optimization Strategies

​
Search-Based Optimizers

These optimizers explore the prompt space systematically:

​
Refinement-Based Optimizers

These optimizers iteratively improve prompts through analysis:
How it works: Analyzes failed examples, formulates hypotheses, and rewrites the entire prompt.Strengths:
  • Deep understanding of failures
  • Holistic prompt redesign
  • Excellent for complex tasks
Limitations:
  • Slower than search-based methods
  • Higher API costs
  • May overfit to evaluation set
How it works: Generates critiques of failures and applies targeted improvements using beam search.Strengths:
  • Systematic error fixing
  • Maintains multiple candidate prompts
  • Good balance of exploration and refinement
Limitations:
  • Can be computationally expensive
  • Requires clear failure signals
  • May need several rounds
How it works: Combines mutation with different “thinking styles”, then critiques and refines top performers.Strengths:
  • Creative exploration
  • Structured refinement process
  • Diverse prompt variations
Limitations:
  • Multiple stages can be slow
  • Requires good teacher model
  • May generate unconventional prompts

​
Evolutionary Optimizers

These use evolutionary strategies inspired by natural selection:
How it works: Uses evolutionary algorithms with reflective learning and mutation strategies.Strengths:
  • State-of-the-art performance
  • Efficient evaluation budgeting
  • Robust to local optima
  • Production-ready
Limitations:
  • Requires external library (gepa)
  • More complex setup
  • Higher computational requirements
Note: GEPA is a powerful external library integrated into our framework.

​
Decision Tree

Do you need production-grade optimization?
├─ Yes → Use GEPA
└─ No
   │
   Do you have few-shot examples in your dataset?
   ├─ Yes → Use Bayesian Search
   └─ No
      │
      Is your task reasoning-heavy or complex?
      ├─ Yes → Use Meta-Prompt
      └─ No
         │
         Do you have clear failure patterns to fix?
         ├─ Yes → Use ProTeGi
         └─ No
            │
            Do you want creative exploration?
            ├─ Yes → Use PromptWizard
            └─ No → Use Random Search (baseline)

​
Combining Optimizers

You can run multiple optimizers sequentially for best results: 
# Stage 1: Quick exploration with Random Search
random_result = random_optimizer.optimize(...)
initial_prompts = [h.prompt for h in random_result.history[:3]]

# Stage 2: Deep refinement with Meta-Prompt
meta_result = meta_optimizer.optimize(
    initial_prompts=initial_prompts,
    ...
)

# Stage 3: Few-shot enhancement with Bayesian Search
final_result = bayesian_optimizer.optimize(
    initial_prompts=[meta_result.best_generator.get_prompt_template()],
    ...
)

​
Next Steps

Try Bayesian Search

Start with the most versatile optimizer

See SDK Guide

Learn how to use optimizers in your code
⌘I