Simulation Testing

Simulation testing lets you run automated conversations against your voice AI agents using synthetic customer personas. For the full platform guide, see Simulation docs. Each simulation produces a transcript, optional audio recordings, and evaluation scores.

Quick Example

import asyncio
from fi.simulate import TestRunner, AgentInput, AgentResponse

runner = TestRunner()

async def my_agent(input: AgentInput) -> str:
    """Your agent logic — receives conversation history, returns a response."""
    user_message = (input.new_message or {}).get("content", "")
    return f"I can help with that: {user_message}"

asyncio.run(runner.run_test(
    run_test_name="basic-test",
    agent_callback=my_agent,
    num_scenarios=3,
))

TestRunner

The main entry point for running simulations.

from fi.simulate import TestRunner

runner = TestRunner(
    api_key="...",       # or FI_API_KEY env var
    secret_key="...",    # or FI_SECRET_KEY env var
)

run_test()

await runner.run_test(
    run_test_name="my-test",
    agent_callback=my_agent,
    num_scenarios=5,
    min_turn_messages=8,
    max_seconds=45.0,
    record_audio=False,
)

Parameter	Type	Default	Description
run_test_name	str	None	Name for this test run
agent_callback	callable	None	Your agent function (cloud mode)
num_scenarios	int	1	Number of conversations to simulate
topic	str or None	None	Topic for auto-generated scenarios
min_turn_messages	int	8	Minimum messages per conversation
max_seconds	float	45.0	Maximum duration per conversation
record_audio	bool	False	Capture audio recordings
scenario	Scenario	None	Pre-defined scenario with personas (local mode)
agent_definition	AgentDefinition	None	Agent config for LiveKit mode

Agent Callback

Your agent receives an AgentInput and returns either a string or an AgentResponse.

from fi.simulate import AgentInput, AgentResponse

# Simple — return a string
async def simple_agent(input: AgentInput) -> str:
    user_msg = (input.new_message or {}).get("content", "")
    # Call your LLM here
    return "Your response"

# Advanced — return AgentResponse with tool calls
async def advanced_agent(input: AgentInput) -> AgentResponse:
    return AgentResponse(
        content="Let me check that for you.",
        tool_calls=[{"name": "lookup_order", "arguments": {"order_id": "12345"}}],
        metadata={"intent": "order_lookup"},
    )

AgentInput

Field	Type	Description
thread_id	str	Conversation ID
messages	list	Full conversation history
new_message	dict or None	Latest user message ({"role": "user", "content": "..."})
execution_id	str or None	Execution tracking ID

AgentResponse

Field	Type	Description
content	str	Agent’s text response
tool_calls	list or None	Tool/function calls made
tool_responses	list or None	Results from tool calls
metadata	dict or None	Custom metadata

Scenarios and Personas

Define custom test scenarios with specific customer personas.

from fi.simulate import Scenario, Persona

scenario = Scenario(
    name="billing-complaints",
    description="Customers with billing issues",
    dataset=[
        Persona(
            persona={"name": "Sarah", "age": 35, "communication_style": "frustrated"},
            situation="Charged twice for the same order",
            outcome="Get a refund and confirmation email",
        ),
        Persona(
            persona={"name": "Mike", "age": 62, "communication_style": "confused"},
            situation="Doesn't understand a charge on the statement",
            outcome="Get a clear explanation of the charge",
        ),
    ],
)

asyncio.run(runner.run_test(
    run_test_name="billing-test",
    scenario=scenario,
    agent_callback=my_agent,
))

Related