A guide to the Meta-Prompt optimizer, which uses a teacher LLM for deep reasoning-based prompt refinement through systematic failure analysis and rewriting.
Meta-Prompt uses a powerful teacher LLM to analyze how your prompt performs, understand why it fails on specific examples, formulate hypotheses about improvements, and completely rewrite the prompt. This approach is inspired by the promptim library and excels at tasks requiring deep reasoning.
{ "hypothesis": "The prompt fails on complex multi-sentence texts because it doesn't specify a structure. Adding explicit instruction to identify main points first should improve clarity.", "improved_prompt": "First identify the 2-3 main points in the following text. Then write a single concise sentence that captures these points:\n\n{text}"}
The Meta-Prompt optimizer is inspired by meta-learning and reflective AI systems, where a model improves its own processes.
Meta-Learning: The core idea is formalized in research like “System Prompt Optimization with Meta-Learning”, which uses bilevel optimization. Another related work is “metaTextGrad”, which optimizes both prompts and their surrounding structures.
Industry Tools: This reflective approach is used in tools like Google’s Vertex AI Prompt Optimizer and is a key feature in advanced models for self-improvement.
Frameworks: The concept is explored in libraries like promptim and is classified in surveys as a leading LLM-driven optimization method.
optimizer = MetaPromptOptimizer( teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}"))result = optimizer.optimize( evaluator=evaluator, data_mapper=data_mapper, dataset=dataset, initial_prompts=[initial_prompt], # Provide detailed context task_description=""" I want to extract structured information from customer support tickets. The prompt should: - Identify the main issue - Extract customer sentiment (positive/negative/neutral) - Determine urgency level (low/medium/high) - Suggest appropriate department routing The output must be in JSON format and handle incomplete information gracefully. """, num_rounds=7, eval_subset_size=30)
# For very complex tasks, use more roundsoptimizer = MetaPromptOptimizer( teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}"))result = optimizer.optimize( evaluator=evaluator, data_mapper=data_mapper, dataset=complex_dataset, initial_prompts=[initial_prompt], task_description=detailed_description, num_rounds=10, # More iterations for complex refinement eval_subset_size=50 # More examples for reliable signal)# Analyze the evolutionfor i, iteration in enumerate(result.history): print(f"\nRound {i+1} Score: {iteration.average_score:.4f}") print(f"Prompt: {iteration.prompt[:150]}...")
Meta-Prompt’s hypotheses show its reasoning process:
Copy
Ask AI
result = optimizer.optimize(...)# View the optimization journeyfor i, iteration in enumerate(result.history): print(f"\n{'='*60}") print(f"Round {i+1}") print(f"Score: {iteration.average_score:.4f}") print(f"\nPrompt:\n{iteration.prompt}") # Note: Hypothesis is internal to teacher model, # but you can infer it from prompt evolution
Meta-Prompt’s quality depends heavily on the teacher model’s reasoning ability. Use gpt-4o, claude-3-opus, or similar high-end models.
Provide detailed task descriptions
Specific task descriptions help the teacher make targeted improvements. Include constraints, desired output format, and edge cases to handle.
Start with 5 rounds
5 rounds is usually enough for meaningful improvement. Increase to 7-10 only for very complex tasks where you see continued progress.
Balance eval subset size
Too small (< 20): Unreliable signal, may optimize for noise
Too large (> 50): Slow and expensive
Sweet spot: 30-40 examples
Analyze failed examples
Look at low-scoring examples in each round to understand what the optimizer is trying to fix:
Copy
Ask AI
for iteration in result.history: failures = [r for r in iteration.individual_results if r.score < 0.5] print(f"Round failures: {len(failures)}") for f in failures[:3]: # Show first 3 print(f" - Score: {f.score:.2f}, Reason: {f.reason}")
optimizer = MetaPromptOptimizer( teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}"))result = optimizer.optimize( evaluator=evaluator, data_mapper=data_mapper, dataset=creative_dataset, initial_prompts=["Write a story based on: {prompt}"], task_description=""" Generate engaging short stories (200-300 words) that: - Have a clear beginning, middle, and end - Include vivid sensory details - Match the tone specified in the prompt - Are appropriate for a general audience """, num_rounds=6, eval_subset_size=25)
optimizer = MetaPromptOptimizer( teacher_generator=LiteLLMGenerator("gpt-4o", "{prompt}"))result = optimizer.optimize( evaluator=evaluator, data_mapper=data_mapper, dataset=transformation_dataset, initial_prompts=["Convert this data: {input_data}"], task_description=""" Transform unstructured text into JSON format with these fields: - name (string) - date (YYYY-MM-DD format) - amount (number) - category (one of: personal, business, travel) Handle missing fields by using null. Infer dates from context when possible. """, num_rounds=5)
