1. Accessing the Optimization Panel
To start optimizing a prompt:- Navigate to the Dataset View – Ensure you have an existing dataset with inputs and model-generated outputs.
- Click on the “Optimize” Button – Located in the top action bar next to Run Prompt, Experiment, and Evaluate.
- Select a Dataset Column – Choose the column containing the prompt you want to optimize.
2. Configuring the Optimization Process
Once you click Optimize, a side panel opens where you define the optimization parameters.a. Naming the Optimization Task
- Enter a meaningful name for the optimization (e.g., Optimize-dataset-1).
- Helps in tracking multiple optimization runs within the Optimization Tab.
b. Selecting the Prompt Column
- Choose the dataset column containing the prompt that needs improvement.
- Ensures that the optimization process is applied to the correct data.
c. Configuring the Prompt
- You can import an existing prompt or define a new prompt template.
- The prompt editor includes:
- System Prompt (Optional) – Provides system-level instructions.
- User Prompt – Defines the primary instruction given to the model.
-
**Column Placeholder
{{column}}**– Dynamically reference dataset column values within the prompt.
3. Selecting the Model and Fine-Tuning Parameters
After defining the prompt, configure the model and its hyperparameters.a. Choose a Language Model
- Select from available LLMs (e.g., GPT-4, GPT-4o-mini, GPT-3.5-turbo).
- Ensure the model aligns with your intended output requirements.
b. Adjust Model Parameters
- Temperature – Controls randomness (higher = more variation, lower = more deterministic).
- Top P – Restricts sampling diversity.
- Max Tokens – Limits the response length.
- Presence & Frequency Penalty – Adjusts repetition patterns.
- Response Format – Defines structured vs. free-text responses.
4. Defining Evaluation Metrics
Evaluation metrics determine how well the optimized prompts perform.a. Adding Preset Evaluations
- Choose from predefined metrics.
- These metrics help the system determine whether the refined prompt performs better than the original.
b. Using Previously Configured Evaluations
- If you have existing evaluation setups, select from the Previously Configured Evals list.
- Ensures consistency across multiple optimizations.
5. Running the Optimization
Once the prompt and evaluation metrics are configured:- Click “Test” – Runs a quick check to verify settings.
- Click “Save & Run” – Starts the optimization process on the dataset.
6. Reviewing Optimization Results
Once optimization is complete, navigate to the Optimization Tab to analyze the results.a. Comparing Optimized Prompts
- The system generates multiple optimized prompt versions.
- The best-performing prompts are ranked based on evaluation scores.
b. Checking Performance Scores
- A table view displays each optimized prompt alongside its evaluation scores.
- Scores for Context Relevance and Context Similarity indicate the effectiveness of each refined prompt.
- The original prompt’s score is included for comparison.
c. Selecting the Best Prompt
- Review the Top 5 optimized templates.
- The system highlights the best-performing prompt based on metrics.
- Users can manually inspect variations before selecting the final prompt.
7. Applying the Optimized Prompt
Once the best-optimized prompt is identified:- Apply the optimized version to replace the original prompt.
- Export the optimized dataset if required.
- Run further iterations if needed to refine performance.
The optimization feature provides a structured way to improve prompt performance. By leveraging evaluation metrics, users can iteratively refine their prompts, ensuring better AI-generated responses. The workflow allows for precise comparison and selection of the most effective prompt for deployment.