Using Platform
This guide outlines the step-by-step process of optimizing prompts using the Future AGI interface. By the end of this guide, you will understand how to configure an optimization task, define evaluation criteria, analyze results, and select the most effective prompt for deployment.
1. Accessing the Optimization Panel
To start optimizing a prompt:
-
Navigate to the Dataset View – Ensure you have an existing dataset with inputs and model-generated outputs.
-
Click on the “Optimize” Button – Located in the top action bar next to Run Prompt, Experiment, and Evaluate.
-
Select a Dataset Column – Choose the column containing the prompt you want to optimize.
2. Configuring the Optimization Process
Once you click Optimize, a side panel opens where you define the optimization parameters.
a. Naming the Optimization Task
- Enter a meaningful name for the optimization (e.g., Optimize-dataset-1).
- Helps in tracking multiple optimization runs within the Optimization Tab.
b. Selecting the Prompt Column
- Choose the dataset column containing the prompt that needs improvement.
- Ensures that the optimization process is applied to the correct data.
c. Configuring the Prompt
- You can import an existing prompt or define a new prompt template.
- The prompt editor includes:
-
System Prompt (Optional) – Provides system-level instructions.
-
User Prompt – Defines the primary instruction given to the model.
-
**Column Placeholder
{{column}}**– Dynamically reference dataset column values within the prompt.
-
3. Selecting the Model and Fine-Tuning Parameters
After defining the prompt, configure the model and its hyperparameters.
a. Choose a Language Model
- Select from available LLMs (e.g., GPT-4, GPT-4o-mini, GPT-3.5-turbo).
- Ensure the model aligns with your intended output requirements.
b. Adjust Model Parameters
-
Temperature – Controls randomness (higher = more variation, lower = more deterministic).
-
Top P – Restricts sampling diversity.
-
Max Tokens – Limits the response length.
-
Presence & Frequency Penalty – Adjusts repetition patterns.
-
Response Format – Defines structured vs. free-text responses.
4. Defining Evaluation Metrics
Evaluation metrics determine how well the optimized prompts perform.
a. Adding Preset Evaluations
- Choose from predefined metrics.
- These metrics help the system determine whether the refined prompt performs better than the original.
b. Using Previously Configured Evaluations
-
If you have existing evaluation setups, select from the Previously Configured Evals list.
-
Ensures consistency across multiple optimizations.
5. Running the Optimization
Once the prompt and evaluation metrics are configured:
-
Click “Test” – Runs a quick check to verify settings.
-
Click “Save & Run” – Starts the optimization process on the dataset.
6. Reviewing Optimization Results
Once optimization is complete, navigate to the Optimization Tab to analyze the results.
a. Comparing Optimized Prompts
- The system generates multiple optimized prompt versions.
- The best-performing prompts are ranked based on evaluation scores.
b. Checking Performance Scores
- A table view displays each optimized prompt alongside its evaluation scores.
- Scores for Context Relevance and Context Similarity indicate the effectiveness of each refined prompt.
- The original prompt’s score is included for comparison.
c. Selecting the Best Prompt
-
Review the Top 5 optimized templates.
-
The system highlights the best-performing prompt based on metrics.
-
Users can manually inspect variations before selecting the final prompt.
7. Applying the Optimized Prompt
Once the best-optimized prompt is identified:
-
Apply the optimized version to replace the original prompt.
-
Export the optimized dataset if required.
-
Run further iterations if needed to refine performance.
The optimization feature provides a structured way to improve prompt performance. By leveraging evaluation metrics, users can iteratively refine their prompts, ensuring better AI-generated responses. The workflow allows for precise comparison and selection of the most effective prompt for deployment.