Chat Simulation: Run Multi-Persona Conversations via SDK

Go to app.futureagi.com → Simulate → Agent Definition → Create agent definition.

The creation wizard has three steps:

Step 1: Basic Info

Field	Value
Agent type	Chat
Agent name	customer-support-bot
Select language	English

Step 2: Configuration

For Chat agents, the only field is Model Used: select your LLM (e.g. gpt-4o-mini). This step is optional.

Step 3: Behaviour

Field	Value
Prompt / Chains	You are a helpful customer support agent for TechStore. You assist customers with orders, returns, and product questions. Always be professional and solution-oriented.
Knowledge Base	(optional) Select a KB if you want grounded responses
Commit Message	Initial support agent prompt

Click Create to save the agent definition as v1.

Go to Simulate → Personas → Create your own persona.

Each persona has sections for Basic Info, Behavioural Settings, Chat Settings, Custom Properties, and Additional Instructions.

Create these three personas (select type Chat for each):

cooperative-customer

Section	Field	Value
Basic Info	Name	cooperative-customer
Basic Info	Description	A patient, friendly customer who provides clear information and follows instructions.
Behavioural	Personality	Friendly and cooperative
Behavioural	Communication Style	Direct and concise
Chat Settings	Tone	neutral
Chat Settings	Verbosity	balanced
Chat Settings	Typo Level	none

frustrated-customer

Section	Field	Value
Basic Info	Name	frustrated-customer
Basic Info	Description	An impatient customer who has already contacted support once. Wants a fast resolution.
Behavioural	Personality	Impatient and direct
Behavioural	Communication Style	Assertive
Chat Settings	Tone	casual
Chat Settings	Verbosity	brief
Chat Settings	Typo Level	occasional

confused-customer

Section	Field	Value
Basic Info	Name	confused-customer
Basic Info	Description	A non-technical user unsure what information to provide. Needs guidance.
Behavioural	Personality	Anxious
Behavioural	Communication Style	Questioning
Chat Settings	Tone	casual
Chat Settings	Verbosity	detailed
Chat Settings	Typo Level	rare

Go to Simulate → Scenarios → Create New Scenario.

Scenarios define the test cases your personas will run against your agent. There are four scenario types:

Type	Use case
Workflow builder	Auto-generate or manually build conversation flows
Import datasets	Use structured data (CSV, JSON, Excel) as test cases
Upload script	Import existing conversation scripts
Call / Chat SOP	Define standard operating procedures for testing

For this guide, select Workflow builder and fill in:

Field	Value
Scenario Name	order-return-request
Description	Customer wants to return a laptop with a cracked screen. Has order number but hasn’t initiated a return yet.
Choose source	Select customer-support-bot (Agent Definition)
Choose version	v1
No. of scenarios	20

Attach personas: In the Persona section, leave the Add by default toggle on — this auto-adds all active personas to your scenarios. Alternatively, turn the toggle off and click Add persona to manually select specific personas.

Click Create.

Go to Simulate → Run Simulation → Create a Simulation.

The creation wizard has four steps:

Step 1: Add simulation details

Field	Value
Simulation name	return-flow-test
Choose Agent definition	customer-support-bot
Choose version	v1
Description	Testing return flow with 3 customer personas

Step 2: Choose Scenario(s)

Select the order-return-request scenario from the list. You can search and select multiple scenarios.

Step 3: Select Evaluations

Click Add Evaluations and under Groups, select Conversational agent evaluation for broad coverage. This group includes 10 built-in evals:

• customer_agent_loop_detection
• customer_agent_context_retention
• customer_agent_query_handling
• customer_agent_termination_handling
• customer_agent_conversation_quality
• customer_agent_objection_handling
• customer_agent_language_handling
• customer_agent_human_escalation
• customer_agent_clarification_seeking
• customer_agent_prompt_conformance

If your agent uses tool calling, toggle Enable tool call evaluation. The platform will automatically evaluate every tool invocation made during the simulation and show Pass/Fail results as additional columns in the results grid (e.g., “check_order_status #1”) with reasoning; no extra code needed.

Step 4: Summary

Review your simulation configuration — agent definition, scenarios, and evaluations — then click Run Simulation to create the simulation.

After the simulation is created, the platform shows SDK instructions with a code snippet to run the simulation. Chat simulations are executed via the SDK — copy the code and proceed to the next step.

Chat simulations require the SDK to execute. The platform generates a code snippet after you create the simulation; replace the placeholder agent with your real agent logic.

import asyncio
import os
import openai
from fi.simulate import TestRunner, AgentInput

openai_client = openai.AsyncOpenAI()

SYSTEM_PROMPT = """You are a helpful customer support agent for TechStore.
You assist customers with orders, returns, and product questions.
Always be professional, empathetic, and solution-oriented.
If you cannot resolve an issue, offer to escalate to a human agent."""


async def agent_callback(input: AgentInput) -> str:
    # Build the full conversation history for context
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    for msg in input.messages:
        messages.append(msg)

    response = await openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0.2,
    )
    return response.choices[0].message.content or ""


async def main():
    runner = TestRunner(
        api_key=os.environ["FI_API_KEY"],
        secret_key=os.environ["FI_SECRET_KEY"],
    )

    report = await runner.run_test(
        run_test_name="return-flow-test",
        agent_callback=agent_callback,
    )

    print(f"Simulation finished! Processed {len(report.results)} test cases")


asyncio.run(main())

Expected output:

🔍 Fetching Run Test ID for name: return-flow-test
✓ Found Run Test ID: <uuid>
Starting Simulation for Run ID: <uuid>
✓ Test Execution Started: <uuid>
🔄 Fetching batch of scenarios...
📥 Received batch: 20 calls
▶️ Processing Call: <uuid>
✓ Call Finished: <uuid> (6 turns)
...
✅ Cloud Simulation Completed.
Simulation finished! Processed 20 test cases

Once the simulation completes, go to Simulate → Run Simulation → open return-flow-test. The results page shows three tabs:

• Chat Details: per-conversation transcripts, CSAT scores, and evaluation scores
• Analytics: evaluation score distributions and trends
• Optimization Runs: results from prompt optimization runs

Fix My Agent: Click the Fix My Agent button (top-right) to open the diagnostic drawer. The platform analyzes your simulation traces and surfaces two categories of recommendations:

• Fixable Recommendations: organized into two tabs:
  • Agent Level: prompt and behavior improvements you can apply directly (e.g. missing empathy phrases, unclear escalation paths)
  • Branch Level: domain-specific issues grouped by conversation topic or flow (e.g. return policy gaps, billing confusion). Each recommendation highlights which specific calls are affected, so you can trace issues back to exact conversations.
• Non-Fixable Recommendations: system-level issues that require infrastructure changes (e.g. missing integrations, data access limitations), plus a human comparison summary showing where a human agent would have handled the situation differently.
• Overall Insights: a synthesis of patterns across all calls.

Optimize My Agent: Inside the Fix My Agent drawer, click Optimize My Agent to generate improved prompt variants automatically:

1. Enter a Name for the optimization run
2. Choose Optimizer — select from available optimizers (e.g. Bayesian Search, MetaPrompt, ProTeGi, GEPA, PromptWizard, Random Search)
3. Language Model — select the model for optimization
4. Click Start Optimizing your agent

Optimization results appear in the Optimization Runs tab. Review the generated prompt variants and their scores to decide which version to promote.