Run a chat simulation

A chat simulation plays every row of its scenarios against your agent, but a chat agent lives in your own code, somewhere Future AGI can’t reach on its own. You write a callback that answers on your agent’s behalf, and the SDK calls it once per turn and carries the reply back into the conversation. This guide creates the run test, then implements that callback with the agent-simulate package.

Create the chat simulation

Build the run test in Create a simulation: name it, pick a chat agent definition and version, tick the chat scenarios you want it to face, and attach evals. The wizard is the same one voice run tests use; choosing a chat agent definition on the first step is what narrows the second step to chat scenarios.

Click Run Simulation on the last step and nothing plays yet. Creating the run test only saves the bundle, since there’s no agent on Future AGI’s side to call. Its Simulated runs tab shows the install and run snippet you’re about to use, already carrying the run test’s exact name.

Write the agent callback

Install the SDK:

pip install agent-simulate

Each turn, the SDK calls your callback with an AgentInput and expects a plain string or an AgentResponse back:

• new_message: the message to answer this turn, shaped {"role": ..., "content": ...}
• messages: the full conversation so far, including that message
• thread_id: identifies which conversation this turn belongs to
• execution_id: the run this call is part of

A callback is either a plain async function or a class extending AgentWrapper; both take an AgentInput and return Union[str, AgentResponse]. my_agent in the examples below stands in for your own agent code; swap it for whatever actually answers your users.

from typing import Union
from fi.simulate import AgentInput, AgentResponse

async def agent_callback(input: AgentInput) -> Union[str, AgentResponse]:
    user_text = input.new_message["content"] if input.new_message else ""
    return await my_agent.respond(user_text)

from typing import Union
from fi.simulate import AgentWrapper, AgentInput, AgentResponse

class MyAgent(AgentWrapper):
    async def call(self, input: AgentInput) -> Union[str, AgentResponse]:
        user_text = input.new_message["content"] if input.new_message else ""
        return await my_agent.respond(user_text)

# pass an instance: agent_callback=MyAgent()

Return a plain string when there’s nothing else to report. Return an AgentResponse when your agent called tools this turn:

• content (required): the reply text
• tool_calls: the tools your agent invoked
• tool_responses: what those tools returned, as {"role": "tool", "tool_call_id": ..., "content": ...} entries
• metadata: anything else you want attached to the turn

return AgentResponse(
    content="Let me check that order for you.",
    tool_calls=[
        {"id": "call_1", "type": "function", "function": {"name": "lookup_order", "arguments": '{"order_id": "123"}'}}
    ],
    tool_responses=[
        {"role": "tool", "tool_call_id": "call_1", "content": '{"status": "shipped"}'}
    ],
)

Scoring those tool calls is a separate opt-in step, covered in Evaluate tool calls.

Run the simulation

Create a TestRunner and point run_test at the run test:

import asyncio
from fi.simulate import TestRunner, AgentInput

async def customer_support_agent(input: AgentInput) -> str:
    user_message = input.new_message["content"] if input.new_message else ""
    return await my_agent.respond(user_message)

async def main():
    runner = TestRunner()
    await runner.run_test(
        run_test_name="Simulating support-agent-chat",  # exact match to the run test's name
        agent_callback=customer_support_agent,
        concurrency=5,
    )
    print("Simulation finished. View results in the dashboard.")

asyncio.run(main())

TestRunner() reads FI_API_KEY and FI_SECRET_KEY from your environment; pass api_key/secret_key directly if you’d rather not use env vars, and api_url (or FI_BASE_URL) if you’re pointing at a self-hosted deployment. Missing credentials don’t fail immediately, they only log a warning, then fail once the SDK actually calls the backend, so check both keys first if a run test stays empty.

run_test takes run_test_name, matched exactly to the run test you created, or run_id if you already have it, plus your agent_callback. concurrency sets how many scenario rows it plays at once.

Run the script and Future AGI plays each scenario row as a persona, turn by turn, against your callback, until every row has either finished or failed.

Dive deeper