Getting Started with Simulations

This comprehensive guide will walk you through creating and running your first AI agent simulation test using the FutureAGI UI. We’ll use an insurance sales agent as our example throughout this guide.

Overview

Creating a simulation test in FutureAGI involves four main steps:

Creating an Agent Definition - Define the AI agent that will handle customer interactions
Creating a Scenario - Set up test cases with sample customer data
Creating a Simulation Agent - Configure an agent that will simulate customer behavior
Running the Test - Execute the simulation and analyze results

Prerequisites

Before you begin, ensure you have:

Access to your FutureAGI dashboard
Basic understanding of AI agents and simulations
Sample customer data for testing (we’ll provide examples)

Step 1: Creating an Agent Definition

The agent definition configures how your AI agent will behave during customer interactions. For our insurance sales agent example, we’ll create an agent that can handle insurance inquiries and sales.

1.1 Navigate to Agent Definitions

From your FutureAGI dashboard, navigate to the Simulations section in the sidebar, then click on Agent Definitions.

1.2 Create New Agent Definition

Click the “Add Agent Definition” button to open the creation dialog.

1.3 Configure Agent Information

Fill in the following fields for your insurance sales agent:

Basic Information

Agent Name: Insurance Sales Agent
Description: AI agent specialized in selling life and health insurance policies to customers

Contact Configuration

Contact Number: +1-800-INSURE-ME (or your test number)
Pin Code: Select your country code (e.g., +1 for US)
Language: English (en)
Connection Type: Toggle to Inbound (agent receives calls)

Provider Settings

Provider: Select provider (e.g., ‘Vapi’ or ‘Retell’)
Assistant ID: Enter your provider-specific assistant ID (optional)

1.4 Save Agent Definition

Click “Create” to save your agent definition. You should see a success message confirming the creation.

Step 2: Creating a Scenario

Scenarios contain the test data and cases that will be used during simulations. For our insurance sales example, we’ll create scenarios with different customer profiles.

2.1 Navigate to Scenarios

From the Simulations section, click on Scenarios.

2.2 Create New Scenario

Click the “Add Scenario” button to open the scenario creation dialog.

2.3 Choose Scenario Type

Select “Dataset” as the scenario type (recommended for structured test data).

2.4 Configure Scenario Details

Basic Information

Scenario Name: Insurance Sales Test Cases
Description: Various customer profiles for testing insurance sales conversations

2.5 Select or Create Dataset

You can either:

Select an existing dataset from the list
Create a new dataset with your test data

Click here to learn how to create a dataset

For this example, sample dataset used here is shown below:

customer_id,name,age,occupation,annual_income,family_status,insurance_interest,budget_monthly,objection_type
CUST001,John Smith,35,Software Engineer,120000,Married with 2 kids,Life Insurance,200-300,Price Sensitive
CUST002,Sarah Johnson,28,Teacher,65000,Single,Health Insurance,150-200,Coverage Concerns
CUST003,Michael Chen,42,Business Owner,150000,Married with 3 kids,Whole Life Insurance,400-500,Trust Issues

Key columns for insurance sales testing:

customer_id: Unique identifier
demographics: name, age, occupation, income
insurance_needs: current coverage, interest, budget
sales_intelligence: objection type, urgency level, communication preference

Comprehensive Test Dataset

For thorough testing, we recommend using our complete sample dataset with 20 diverse customer profiles that includes:

Various Demographics: Ages 26-55, incomes $50k-$ 250k
Different Occupations: Healthcare, tech, education, business owners
Family Situations: Single, married, divorced, with/without dependents
Health Conditions: From none to chronic conditions
Insurance Interests: Life, health, disability, supplemental coverage
Objection Types: Price sensitivity, trust issues, coverage concerns, time constraints
Risk Profiles: Low, medium, and high-risk customers

This variety ensures your agent is tested against realistic customer scenarios.

2.6 Save Scenario

Click “Create” to save your scenario. The system will validate your dataset and create the scenario.

Step 3: Creating a Simulation Agent

The simulation agent simulates customer behavior during the test. It will interact with your insurance sales agent based on the scenario data.

3.1 Navigate to Simulation Agents

From the Simulations section, click on Simulation Agents.

3.2 Create New Simulation Agent

Click “Add Simulation Agent” to create a new simulator agent.

3.3 Configure Simulation Agent

Basic Information

Agent Name: Enter a name for the simulation agent (e.g., ‘Insurance Customer Simulator’)
Agent Type: Choose between ‘voice’ or ‘chat’
Prompt: Enter the prompt for the simulation agent
Language Model: Select the language model for the simulation agent (eg. ‘gpt’, ‘claude’ or your custom model)
LLM Temperature: Set the temperature for the language model (0.0 to 1.0) (default: 0.7)

Voice and Speech

Voice Provider: Choose between ElevenLabs, Azure Cognitive Services, Google Cloud, etc.
Voice Name: Choose a voice that suits your use case.
Interrupt Sensitivity: Set the interrupt sensitivity for the voice
Conversation Speed: Set the conversation speed for the voice
Finished Speaking Sensitivity: Set the finished speaking sensitivity for the voice

Conversation

Max call duration (in minutes): Set the maximum call duration for the simulation agent
Initial message delay (in seconds): Set the initial message delay for the simulation agent
Initial message: Set the initial message for the simulation agent

3.4 Save Simulation Agent

Click “Create” to save your simulation agent configuration.

Step 4: Creating and Running a Test

Now we’ll combine all components to create and run a simulation test.

4.1 Navigate to Run Tests

From left sidebar, click on Run Tests.

4.2 Create New Test

Click “Create Test” to start the test creation wizard.

4.3 Test Configuration (Step 1 of 5)

Basic Information

Test Name: Insurance Sales Agent Performance Test
Description: Testing the insurance sales agent's ability to handle different customer profiles and close sales

4.4 Select Test Scenarios (Step 2 of 5)

Select the scenario we created earlier:

✓ Insurance Sales Test Cases

You can select multiple scenarios if needed. The test will run through all selected scenarios.

4.5 Select Test Agent (Step 3 of 5)

Choose your insurance sales agent:

✓ Insurance Sales Agent

4.6 Select Evaluations (Step 4 of 5)

Choose evaluation metrics relevant to insurance sales:

✓ Conversation Quality - Measures professionalism and clarity
✓ Product Knowledge - Evaluates accuracy of insurance information
✓ Sales Effectiveness - Tracks conversion and objection handling
✓ Compliance - Ensures regulatory requirements are met
✓ Customer Satisfaction - Simulated CSAT score

4.7 Review Summary (Step 5 of 5)

Review your test configuration:

Test Summary:

Name: Insurance Sales Agent Performance Test
Agent: Insurance Sales Agent
Simulation Agent: Insurance Customer Simulator
Scenarios: 1 scenario with 2 test cases
Evaluations: 5 metrics selected
Estimated Duration: ~20 minutes

4.8 Run the Test

Click “Run Test” to start the simulation.

Step 5: Monitoring Test Execution

5.1 Test Progress

Once the test starts, you’ll see a real-time progress view:

The progress view shows:

Overall Progress: Percentage of test cases completed
Current Scenario: Active test case being executed
Live Metrics: Real-time evaluation scores
Call Logs: Streaming conversation transcripts

5.2 View Test Results

After completion, click on your test to view detailed results:

5.3 Analyze Call Logs

Click on “Call Logs” tab to review individual conversations:

Each call log includes:

Full conversation transcript
Timestamp and duration
Individual evaluation scores
Key moments flagged by evaluations
Audio recording (if enabled)

5.4 Export Results

Export your test results for further analysis:

PDF Report: Comprehensive test summary
CSV Data: Raw evaluation scores
Call Recordings: Audio files (if enabled)

Best Practices

For Insurance Sales Agents

Scenario Diversity: Include various customer profiles:
- Different age groups and income levels
- Various insurance needs (life, health, auto)
- Different objection types
Compliance Testing: Always include compliance evaluations to ensure:
- Proper disclosures are made
- Regulatory requirements are met
- No misleading information is provided
Iterative Improvement:
- Run tests regularly
- Analyze failed conversions
- Update agent prompts based on results

General Tips

Start Small: Begin with 5-10 test cases before scaling up
Use Realistic Data: Base scenarios on actual customer profiles
Monitor Trends: Track performance over multiple test runs
Collaborate: Share results with your team for insights

Troubleshooting

Common Issues

Test Won’t Start

Verify all components (agent, scenarios, simulation agent) are properly configured
Check that your agent’s API credentials are valid
Ensure you have sufficient credits/quota

Low Evaluation Scores

Review agent prompts and instructions
Analyze call logs for specific failure points
Adjust evaluation thresholds if too strict

Timeout Errors

Reduce scenario complexity
Increase timeout settings in test configuration
Check agent response time settings

Next Steps

Now that you’ve run your first simulation test:

Iterate and Improve: Use test results to refine your agent
Scale Testing: Add more scenarios and edge cases
Automate: Set up scheduled test runs
Advanced Features: Explore multi-agent simulations and custom evaluations

For more advanced topics, see:

Get Started

Guides

​Getting Started with Simulations

​Overview

​Prerequisites

​Step 1: Creating an Agent Definition

​1.1 Navigate to Agent Definitions

​1.2 Create New Agent Definition

​1.3 Configure Agent Information

​Basic Information

​Contact Configuration

​Provider Settings

​1.4 Save Agent Definition

​Step 2: Creating a Scenario

​2.1 Navigate to Scenarios

​2.2 Create New Scenario

​2.3 Choose Scenario Type

​2.4 Configure Scenario Details

​Basic Information

​2.5 Select or Create Dataset

​Comprehensive Test Dataset

​2.6 Save Scenario

​Step 3: Creating a Simulation Agent

​3.1 Navigate to Simulation Agents

​3.2 Create New Simulation Agent

​3.3 Configure Simulation Agent

​Basic Information

​Voice and Speech

​Conversation

​3.4 Save Simulation Agent

​Step 4: Creating and Running a Test

​4.1 Navigate to Run Tests

​4.2 Create New Test

​4.3 Test Configuration (Step 1 of 5)

​Basic Information

​4.4 Select Test Scenarios (Step 2 of 5)

​4.5 Select Test Agent (Step 3 of 5)

​4.6 Select Evaluations (Step 4 of 5)

​4.7 Review Summary (Step 5 of 5)

​4.8 Run the Test

​Step 5: Monitoring Test Execution

​5.1 Test Progress

​5.2 View Test Results

​5.3 Analyze Call Logs

​5.4 Export Results

​Best Practices

​For Insurance Sales Agents

​General Tips

​Troubleshooting

​Common Issues

​Next Steps

Getting Started with Simulations

Overview

Prerequisites

Step 1: Creating an Agent Definition

1.1 Navigate to Agent Definitions

1.2 Create New Agent Definition

1.3 Configure Agent Information

Basic Information

Contact Configuration

Provider Settings

1.4 Save Agent Definition

Step 2: Creating a Scenario

2.1 Navigate to Scenarios

2.2 Create New Scenario

2.3 Choose Scenario Type

2.4 Configure Scenario Details

Basic Information

2.5 Select or Create Dataset

Comprehensive Test Dataset

2.6 Save Scenario

Step 3: Creating a Simulation Agent

3.1 Navigate to Simulation Agents

3.2 Create New Simulation Agent

3.3 Configure Simulation Agent

Basic Information

Voice and Speech

Conversation

3.4 Save Simulation Agent

Step 4: Creating and Running a Test

4.1 Navigate to Run Tests

4.2 Create New Test

4.3 Test Configuration (Step 1 of 5)

Basic Information

4.4 Select Test Scenarios (Step 2 of 5)

4.5 Select Test Agent (Step 3 of 5)

4.6 Select Evaluations (Step 4 of 5)

4.7 Review Summary (Step 5 of 5)

4.8 Run the Test

Step 5: Monitoring Test Execution

5.1 Test Progress

5.2 View Test Results

5.3 Analyze Call Logs

5.4 Export Results

Best Practices

For Insurance Sales Agents

General Tips

Troubleshooting

Common Issues

Next Steps