Chat Simulation with Personas

Go to app.futureagi.com → Simulate → Agent Definition → Create agent definition.

The creation wizard has three steps:

Step 1: Basic Info

Field	Value
Agent type	Chat
Agent name	customer-support-bot
Select language	English

Step 2: Configuration

For Chat agents, the only field is Model Used: select your LLM (e.g. gpt-4o-mini). This step is optional.

Step 3: Behaviour

Field	Value
Prompt / Chains	You are a helpful customer support agent for TechStore. You assist customers with orders, returns, and product questions. Always be professional and solution-oriented.
Knowledge Base	(optional) Select a KB if you want grounded responses
Commit Message	Initial support agent prompt

Click Create to save the agent definition as v1.

Go to Simulate → Personas → Create your own persona.

Each persona has sections for Basic Info, Behavioural Settings, Chat Settings, Custom Properties, and Additional Instructions.

Create these three personas (select type Chat for each):

cooperative-customer

Section	Field	Value
Basic Info	Name	cooperative-customer
Basic Info	Description	A patient, friendly customer who provides clear information and follows instructions
Behavioural	Personality	Friendly and cooperative
Behavioural	Communication Style	Direct and concise
Chat Settings	Tone	neutral
Chat Settings	Verbosity	balanced
Chat Settings	Typo Level	none

frustrated-customer

Section	Field	Value
Basic Info	Name	frustrated-customer
Basic Info	Description	An impatient customer who has already contacted support once and wants a fast resolution
Behavioural	Personality	Impatient and direct
Behavioural	Communication Style	Assertive
Chat Settings	Tone	casual
Chat Settings	Verbosity	brief
Chat Settings	Typo Level	occasional

confused-customer

Section	Field	Value
Basic Info	Name	confused-customer
Basic Info	Description	A non-technical customer unsure what information to provide, needs guidance
Behavioural	Personality	Anxious
Behavioural	Communication Style	Questioning
Chat Settings	Tone	casual
Chat Settings	Verbosity	detailed
Chat Settings	Typo Level	rare

All persona options:

Section	Field	Options
Behavioural	Personality	Friendly and cooperative, Professional and formal, Cautious and skeptical, Impatient and direct, Detail-oriented, Easy-going, Anxious, Confident, Analytical, Emotional, Reserved, Talkative
Behavioural	Communication Style	Direct and concise, Detailed and elaborate, Casual and friendly, Formal and polite, Technical, Simple and clear, Questioning, Assertive, Passive, Collaborative
Chat Settings	Tone	formal / neutral / casual
Chat Settings	Verbosity	brief / balanced / detailed
Chat Settings	Regional Mix	none / light / moderate / heavy
Chat Settings	Slang Level	none / light / moderate / heavy
Chat Settings	Typo Level	none / rare / occasional / frequent
Chat Settings	Punctuation Style	clean / minimal / expressive / erratic
Chat Settings	Emoji Frequency	never / light / regular / heavy

You can also set Custom Properties (key-value pairs) and Additional Instructions (free text) for more nuanced behavior.

Go to Simulate → Scenarios → Create New Scenario.

Scenarios define the test cases your personas will run against your agent. There are four scenario types:

Type	Use case
Workflow builder	Auto-generate or manually build conversation flows
Import datasets	Use structured data (CSV, JSON, Excel) as test cases
Upload Script	Import existing conversation scripts
Call/Chat SOP	Define standard operating procedures for testing

For this guide, select Workflow builder and fill in:

Field	Value
Scenario Name	order-return-request
Description	Customer wants to return a laptop with a cracked screen, has an order number but hasn’t initiated a return yet
Choose source	Select customer-support-bot (Agent Definition)
Choose version	v1
No. of scenarios	20

Attach personas: in the Persona section, leave the Add by default toggle on to auto-add all active personas to your scenarios. Alternatively, turn the toggle off and click Add persona to manually select specific personas.

Click Create.

You can also add Columns (custom inputs like order IDs, product names, or issue categories) to generate more varied scenario data, and use the Custom Instructions toggle to provide extra context for scenario generation beyond the agent definition.

Go to Simulate → Run Simulation → Create a Simulation.

The creation wizard has four steps:

Step 1: Add simulation details

Field	Value
Simulation name	return-flow-test
Choose Agent definition	customer-support-bot
Choose version	v1
Description	Testing return flow with 3 customer personas

Step 2: Choose Scenario(s)

Select the order-return-request scenario from the list. You can search and select multiple scenarios.

Step 3: Select Evaluations

Click Add Evaluations and under Groups, select Conversational agent evaluation for broad coverage. This group includes 10 built-in evals:

• customer_agent_loop_detection
• customer_agent_context_retention
• customer_agent_query_handling
• customer_agent_termination_handling
• customer_agent_conversation_quality
• customer_agent_objection_handling
• customer_agent_language_handling
• customer_agent_human_escalation
• customer_agent_clarification_seeking
• customer_agent_prompt_conformance

If your agent uses tool calling, toggle Enable tool call evaluation. The platform automatically evaluates every tool invocation made during the simulation and shows Pass/Fail results as additional columns in the results grid (e.g. “check_order_status #1”) with reasoning, no extra code needed.

Step 4: Summary

Review your simulation configuration (agent definition, scenarios, and evaluations), then click Run Simulation to create the simulation.

After the simulation is created, the platform shows SDK instructions with a code snippet to run the simulation. Chat simulations run from the SDK. Copy the code and continue to the next step.

Chat simulations require the SDK to execute. The platform generates a code snippet after you create the simulation; replace the placeholder agent with your real agent logic.

import asyncio
import os
import openai
from fi.simulate import TestRunner, AgentInput

openai_client = openai.AsyncOpenAI()

SYSTEM_PROMPT = """You are a helpful customer support agent for TechStore.
You assist customers with orders, returns, and product questions.
Always be professional, empathetic, and solution-oriented.
If you cannot resolve an issue, offer to escalate to a human agent."""


async def agent_callback(input: AgentInput) -> str:
    # Build the full conversation history for context
    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
    for msg in input.messages:
        messages.append(msg)

    response = await openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0.2,
    )
    return response.choices[0].message.content or ""


async def main():
    runner = TestRunner(
        api_key=os.environ["FI_API_KEY"],
        secret_key=os.environ["FI_SECRET_KEY"],
    )

    await runner.run_test(
        run_test_name="return-flow-test",
        agent_callback=agent_callback,
    )

    print("Transcripts and eval scores are in the return-flow-test run in Simulate")


asyncio.run(main())

Expected output:

🔍 Fetching Run Test ID for name: return-flow-test
✓ Found Run Test ID: <uuid>
Starting Simulation for Run ID: <uuid>
✓ Test Execution Started: <uuid>
🔄 Fetching batch of scenarios...
📥 Received batch: 20 calls
▶️ Processing Call: <uuid>
✓ Call Finished: <uuid> (6 turns)
...
✅ Cloud Simulation Completed.

Once the simulation completes, go to Simulate → Run Simulation → open return-flow-test. The results page shows three tabs:

• Chat Details: per-conversation transcripts, CSAT scores, and evaluation scores
• Analytics: evaluation score distributions and trends
• Optimization Runs: results from prompt optimization runs

Fix My Agent: click the Fix My Agent button (top-right) to open the diagnostic drawer. The platform analyzes your simulation traces and surfaces two categories of recommendations:

• Fixable Recommendations, organized into two tabs:
  • Agent Level: prompt and behavior improvements you can apply directly (e.g. missing empathy phrases, unclear escalation paths)
  • Branch Level: domain-specific issues grouped by conversation topic or flow (e.g. return policy gaps, billing confusion). Each recommendation highlights which specific calls are affected, so you can trace issues back to exact conversations
• Non-Fixable Recommendations: system-level issues that require infrastructure changes (e.g. missing integrations, data access limitations), plus a human comparison summary showing where a human agent would have handled the situation differently
• Overall Insights: a synthesis of patterns across all calls

For example (figures below are illustrative, not measured from a captured run): the frustrated-customer conversations are where a customer_agent_human_escalation failure is likeliest to show up, with the drawer’s reason string reading something like “agent did not offer escalation after the customer expressed repeated dissatisfaction,” scoring 0 on that eval. The matching Agent Level recommendation adds an explicit escalation offer to the prompt after two failed resolution attempts; applying it and rerunning the scenario would move that same eval from 0 to 1 on the next simulation.

Optimize My Agent: inside the Fix My Agent drawer, click Optimize My Agent to generate improved prompt variants automatically:

1. Enter a Name for the optimization run
2. Choose Optimizer: select from available optimizers (e.g. Bayesian Search, MetaPrompt, ProTeGi, GEPA, PromptWizard, Random Search)
3. Language Model: select the model for optimization
4. Click Start Optimizing your agent

Optimization results appear in the Optimization Runs tab. Review the generated prompt variants and their scores to decide which version to promote.