Simulation Testing: AI Agent Testing SDK Module

Simulation testing lets you run automated conversations against your AI agents using synthetic customer personas. For the full platform guide, see Simulation docs. Each simulation produces a transcript and evaluation scores, plus audio recordings when you run the local voice mode.

Two modes

TestRunner.run_test() is one method that picks its mode from the arguments you pass.

Mode	Selected by	What happens	Requires
Cloud	run_id or run_test_name	Future AGI orchestrates the simulated customer and calls your agent_callback each turn	FI_API_KEY and FI_SECRET_KEY
Local voice	agent_definition	A simulated customer joins your agent’s LiveKit room over WebRTC	agent-simulate[livekit] and a running LiveKit deployment

Passing neither raises ValueError. The two argument sets do not combine: if you pass both, run_id or run_test_name wins and the local-mode arguments are ignored.

Quick Example

import asyncio
from fi.simulate import TestRunner, AgentInput, AgentResponse

runner = TestRunner()

async def my_agent(input: AgentInput) -> str:
    """Your agent logic — receives conversation history, returns a response."""
    user_message = (input.new_message or {}).get("content", "")
    return f"I can help with that: {user_message}"

asyncio.run(runner.run_test(
    run_test_name="basic-test",
    agent_callback=my_agent,
))

run_test_name must match the name of a run test you created on the platform. Pass run_id instead if you already have its ID.

TestRunner

The main entry point for running simulations.

from fi.simulate import TestRunner

runner = TestRunner(
    api_key="...",       # or FI_API_KEY env var
    secret_key="...",    # or FI_SECRET_KEY env var
)

run_test()

Cloud mode:

await runner.run_test(
    run_test_name="my-test",   # or run_id="<uuid>"
    agent_callback=my_agent,
    concurrency=5,
)

Local voice mode, which needs the [livekit] extra:

await runner.run_test(
    agent_definition=agent,    # your deployed agent's LiveKit room
    scenario=scenario,
    record_audio=True,
    max_seconds=45.0,
)

Parameter	Type	Default	Mode	Description
run_id	str or None	None	Cloud	ID of the run test to execute
run_test_name	str or None	None	Cloud	Name of the run test, resolved to an ID for you; use instead of run_id
agent_callback	callable or AgentWrapper	None	Cloud	Your agent function or wrapper instance
concurrency	int	5	Cloud	How many calls run in parallel
agent_definition	AgentDefinition	None	Local	The deployed voice agent to dial into
scenario	Scenario or None	None	Local	Pre-defined scenario with personas
simulator	SimulatorAgentDefinition or None	None	Local	Overrides the simulated customer’s LLM, TTS, STT, and VAD settings
num_scenarios	int	1	Local	Scenarios to generate when scenario is omitted
topic	str or None	None	Local	Topic for auto-generated scenarios
record_audio	bool	False	Local	Capture per-speaker and combined WAV files
min_turn_messages	int	8	Local	Minimum messages per conversation
max_seconds	float	45.0	Local	Maximum duration per conversation

Agent Callback

Your agent receives an AgentInput and returns either a string or an AgentResponse.

from fi.simulate import AgentInput, AgentResponse

# Simple — return a string
async def simple_agent(input: AgentInput) -> str:
    user_msg = (input.new_message or {}).get("content", "")
    # Call your LLM here
    return "Your response"

# Advanced — return AgentResponse with tool calls
async def advanced_agent(input: AgentInput) -> AgentResponse:
    return AgentResponse(
        content="Let me check that for you.",
        tool_calls=[{"name": "lookup_order", "arguments": {"order_id": "12345"}}],
        metadata={"intent": "order_lookup"},
    )

AgentInput

Field	Type	Description
thread_id	str	Conversation ID
messages	list	Full conversation history
new_message	dict or None	Latest user message ({"role": "user", "content": "..."})
execution_id	str or None	Execution tracking ID

AgentResponse

Field	Type	Description
content	str	Agent’s text response
tool_calls	list or None	Tool/function calls made
tool_responses	list or None	Results from tool calls
metadata	dict or None	Custom metadata

Scenarios and Personas

A Scenario is a named list of personas. It is a local voice mode argument: in cloud mode the personas come from the run test you configured on the platform, and a scenario passed to run_test is ignored.

from fi.simulate import AgentDefinition, Scenario, Persona

agent = AgentDefinition(
    name="billing-support-agent",
    url="wss://your-livekit-server.com",
    room_name="support-room",
    system_prompt="You are a helpful billing support agent.",
)

scenario = Scenario(
    name="billing-complaints",
    description="Customers with billing issues",
    dataset=[
        Persona(
            persona={"name": "Sarah", "age": 35, "communication_style": "frustrated"},
            situation="Charged twice for the same order",
            outcome="Get a refund and confirmation email",
        ),
        Persona(
            persona={"name": "Mike", "age": 62, "communication_style": "confused"},
            situation="Doesn't understand a charge on the statement",
            outcome="Get a clear explanation of the charge",
        ),
    ],
)

asyncio.run(runner.run_test(
    agent_definition=agent,
    scenario=scenario,
))

Related