Chat Simulation Using SDK

Go to Simulate → Run Simulation → Create a Simulation. Use a chat agent definition and version, add scenarios and optional evals, then save. Open the simulation from Simulate → Run Simulation by clicking it — you’re on the simulation detail (e.g. Simulated runs tab). For full setup (agent, scenarios, personas), see Run simulation.

On the simulation detail page, click Run New Simulation. For chat agent simulations (non–prompt), the UI does not call the execute API; it opens a right-side drawer with SDK instructions: Step 1 — install the SDK (copy/run the snippet); Step 2 — create a simulation run (copy/run the code to start the simulation from your environment). You can use that code as-is or follow the steps below. The drawer content comes from the same install and run snippets described in this guide.

pip install agent-simulate litellm

litellm is optional; use it if you want to call OpenAI/Anthropic/Gemini from the example. Set FI_API_KEY and FI_SECRET_KEY (env vars or pass into TestRunner). If your callback calls an LLM, set the provider key (e.g. OPENAI_API_KEY).

Your callback receives AgentInput (each turn the simulator sends thread_id, messages, new_message, execution_id) and returns a string or AgentResponse (content, and optionally tool_calls, tool_responses, metadata). You can use a plain async function or the AgentWrapper class — implement async def call(self, input: AgentInput) -> Union[str, AgentResponse] and pass an instance as agent_callback.

• input.new_message — The latest simulator message you should respond to (the “user” message for this turn).
• input.messages — Full conversation history so far (including the latest simulator message).
• input.thread_id / input.execution_id — For logging or correlation.

If your agent uses tools, return an AgentResponse with content, tool_calls, and (if you have them) tool_responses; you can mock tool outputs inside the callback.

async def agent_callback(input: AgentInput) -> Union[str, AgentResponse]:
    user_text = (input.new_message or {}).get("content", "") or ""
    # Call your LLM or logic; return str or AgentResponse
    return "Your reply"

from fi.simulate import AgentWrapper, AgentInput, AgentResponse
from typing import Union

class MyAgent(AgentWrapper):
    async def call(self, input: AgentInput) -> Union[str, AgentResponse]:
        user_text = (input.new_message or {}).get("content", "") or ""
        return f"You said: {user_text}"

# await runner.run_test(..., agent_callback=MyAgent(), ...)

return AgentResponse(
    content="Let me look that up.",
    tool_calls=[{"id": "call_1", "type": "function", "function": {"name": "lookup_order", "arguments": '{"order_id": "123"}'}}],
    tool_responses=[{"role": "tool", "tool_call_id": "call_1", "content": '{"status": "shipped"}'}],
)

Create a TestRunner with your API key and secret, then call run_test with the exact simulation name (the name shown in Simulate → Run Simulation) and your callback. Run this code in your terminal or script — the SDK talks to the backend and creates/runs the simulation; results then show under the same simulation (e.g. Simulated runs tab):

from fi.simulate import TestRunner, AgentInput, AgentResponse
import litellm
import os
from typing import Union
import asyncio

FI_API_KEY = os.environ.get("FI_API_KEY", "<YOUR_FI_API_KEY>")
FI_SECRET_KEY = os.environ.get("FI_SECRET_KEY", "<YOUR_FI_SECRET_KEY>")
run_test_name = "Chat test"  # must match simulation name in UI (Simulate → Run Simulation)
concurrency = 5

async def agent_callback(input: AgentInput) -> Union[str, AgentResponse]:
    user_text = (input.new_message or {}).get("content", "") or ""
    resp = await litellm.acompletion(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": user_text}],
        temperature=0.2,
    )
    return resp.choices[0].message.content or ""

async def main():
    runner = TestRunner(api_key=FI_API_KEY, secret_key=FI_SECRET_KEY)
    await runner.run_test(
        run_test_name=run_test_name,
        agent_callback=agent_callback,
        concurrency=concurrency,
    )
    print("Simulation completed. View results in the dashboard.")

asyncio.run(main())

Transcripts, metrics, and evaluations appear under the same simulation in the dashboard (e.g. Simulated runs tab). Open the simulation → select the test execution → open a call execution to see the full transcript and eval results. The SDK orchestrates runs and supplies agent replies; the platform stores all results.