When to Use Random Search
✅ Best For
- Establishing a quick baseline
- Simple tasks like summarization or classification
- Broad, unbiased exploration of the prompt space
- Projects with a low computational budget
❌ Not Ideal For
- Complex, nuanced, or multi-step reasoning tasks
- Directed, efficient optimization when failure modes are known
- Tasks requiring highly structured or constrained prompts
- Finding the absolute, state-of-the-art best prompt
How It Works
The Random Search process is simple and effective, involving three main steps:1
1. Generate Variations
You provide an initial prompt. The optimizer then uses a powerful
teacher_model (like GPT-4o) to generate a specified num_variations of diverse rewrites of that prompt.2
2. Evaluate All Variations
The optimizer iterates through each generated variation. For each one, it generates outputs for all examples in your dataset and scores them using the provided evaluator.
3
3. Select the Best
The variation that achieves the highest average score across the entire dataset is chosen as the best prompt. The process concludes, and this top-performing prompt is returned.
Basic Usage
Configuration Parameters
The generator instance that you want to optimize. The optimizer will modify the prompt template within this object.
The powerful language model used to generate the prompt variations. The quality of the random search depends heavily on this model’s ability to create diverse and sensible rewrites. Recommended:
gpt-4o, claude-3-opus.The number of different prompt variations the teacher model will generate. This parameter controls the trade-off between the breadth of the search and the computational cost/time of the optimization.
A dictionary of additional arguments to pass to the teacher model during variation generation. This is useful for controlling parameters like
temperature to influence the creativity of the variations.Underlying Research
Random search is a foundational technique in hyperparameter tuning, valued for its simplicity and surprising effectiveness, often outperforming more structured methods like grid search.- Baseline Strength: Research like “Random Sampling as a Strong Baseline for Prompt Optimisation” demonstrates that even simple random sampling can be a highly competitive method for improving prompts.
- Broad Applicability: It is frequently used as the first step in prompt optimization toolkits to get a sense of the landscape. Its ability to avoid getting stuck in local optima makes it a valuable tool in the discrete and high-dimensional space of prompt engineering.