Test a Local Voice Agent

Copy the API key, secret, and URL the livekit-server command printed. LiveKit’s real-time SDK requires the ws:// scheme, not http://.

import os
import getpass

os.environ["LIVEKIT_URL"] = "ws://127.0.0.1:7880"
os.environ["LIVEKIT_API_KEY"] = "devkey"  # From livekit-server output
os.environ["LIVEKIT_API_SECRET"] = "secret"  # From livekit-server output
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")

os.environ["FI_API_KEY"] = getpass.getpass("Enter your FI API key: ")
os.environ["FI_SECRET_KEY"] = getpass.getpass("Enter your FI secret key: ")

You should see three password prompts, then no errors. FI_API_KEY and FI_SECRET_KEY are only read later, in Step 4.

Instead of pointing at a deployed agent, define a SupportAgent locally. You’ll start this function yourself as a background task, right before calling TestRunner.run_test, so it’s already connected to the LiveKit room when the simulated customer joins.

import asyncio
import logging

from fi.simulate import AgentDefinition, Scenario, Persona, TestRunner
from livekit import rtc
from livekit.api import AccessToken, VideoGrants
from livekit.agents import Agent, AgentSession, function_tool
from livekit.plugins import openai, silero
from livekit.agents.voice.room_io import RoomInputOptions, RoomOutputOptions

logging.basicConfig(level=logging.INFO)

class SupportAgent(Agent):
    def __init__(self, *, room: rtc.Room, **kwargs):
        super().__init__(**kwargs)
        self._room = room

    @function_tool()
    async def end_call(self) -> None:
        self.session.say("I'm glad I could help. Have a great day! Goodbye.")
        await asyncio.sleep(0.2)
        self.session.shutdown()
        if self._room.isconnected():
            await self._room.disconnect()

async def run_support_agent(lk_url: str, lk_api_key: str, lk_api_secret: str, room_name: str):
    token = (
        AccessToken(lk_api_key, lk_api_secret)
        .with_identity("support-agent")
        .with_grants(VideoGrants(room_join=True, room=room_name))
        .to_jwt()
    )
    room = rtc.Room()
    await room.connect(lk_url, token)

    agent = SupportAgent(
        room=room,
        stt=openai.STT(),
        llm=openai.LLM(model="gpt-4o-mini", temperature=0.7),
        tts=openai.TTS(voice="alloy"),
        vad=silero.VAD.load(),
        allow_interruptions=True,
        instructions=(
            "You are a helpful support agent. Be friendly and proactive. "
            "Ask clarifying questions and provide step-by-step guidance. "
            "When the customer confirms their issue is resolved, "
            "call the `end_call` tool to gracefully end the call."
        ),
    )

    session = AgentSession(
        stt=agent.stt,
        llm=agent.llm,
        tts=agent.tts,
        vad=None,
        turn_detection="stt",
        allow_interruptions=True,
    )
    await session.start(
        agent,
        room=room,
        room_input_options=RoomInputOptions(delete_room_on_close=False),
        room_output_options=RoomOutputOptions(transcription_enabled=False),
    )

    await asyncio.sleep(0.6)
    session.say("Hello! How can I help you today?")

    closed = asyncio.Event()
    session.on("close", lambda ev: closed.set())
    await closed.wait()
    if room.isconnected():
        await room.disconnect()

You should see no output yet. This defines the agent function you start below.

A Scenario holds one or more Persona objects, each describing a simulated customer’s situation and the outcome the agent should reach.

room_name = "test-room-1"

agent_definition = AgentDefinition(
    name="deployed-support-agent",
    url=os.environ["LIVEKIT_URL"],
    room_name=room_name,
    system_prompt="Helpful support agent",
)

scenario = Scenario(
    name="Account Login Support",
    dataset=[
        Persona(
            persona={"name": "Fubar", "mood": "annoyed"},
            situation="He is trying to log into his account but keeps getting an 'invalid password' error, even though he's sure it's correct.",
            outcome="The agent should calmly guide him to reset his password.",
        ),
    ],
)

You should see no output. agent_definition and scenario are the two arguments TestRunner.run_test needs next.

TestRunner creates a LiveKit room, connects the simulated customer, and records the conversation. Start run_support_agent as a background task first, so it’s already in the room when run_test starts the simulated customer:

support_task = asyncio.create_task(
    run_support_agent(
        os.environ["LIVEKIT_URL"],
        os.environ["LIVEKIT_API_KEY"],
        os.environ["LIVEKIT_API_SECRET"],
        room_name,
    )
)

runner = TestRunner()
report = await runner.run_test(
    agent_definition,
    scenario,
    record_audio=True,
    max_seconds=240.0,
)

print(report.model_dump_json(indent=2))

You should see a TestReport JSON blob with one result per persona in scenario.dataset. The run can take a few minutes since it plays out the full voice conversation.

Each result on the report carries the full transcript and a path to the recorded audio.

for result in report.results:
    print("--- Transcript ---")
    print(result.transcript)
    if result.audio_combined_path and os.path.exists(result.audio_combined_path):
        print(f"Audio file saved at: {result.audio_combined_path}")

You should see the back-and-forth between “Fubar” and the support agent, ending with the agent walking through a password reset. In a notebook, play the audio file with IPython.display.Audio(result.audio_combined_path).

evaluate_report runs fi.evals templates against fields on the TestReport. Each eval_specs entry maps a template to the report fields it needs.

from fi.simulate.evaluation import evaluate_report

eval_specs = [
    {"template": "task_completion", "map": {"input": "persona.situation", "output": "transcript"}},
    {"template": "tone", "map": {"output": "transcript"}},
    {"template": "is_harmful_advice", "map": {"output": "transcript"}},
    {"template": "answer_refusal", "map": {"input": "persona.situation", "output": "transcript"}},
]

report = evaluate_report(
    report,
    eval_specs=eval_specs,
    model_name="turing_large",
    api_key=os.environ["FI_API_KEY"],
    secret_key=os.environ["FI_SECRET_KEY"],
)

for result in report.results:
    print(f"--- Persona: {result.persona.persona['name']} ---")
    if result.evaluation:
        for k, v in result.evaluation.items():
            print(f"  - {k}: {v}")

You should see an evaluation result per template on each persona result, for example a task_completion score. Illustrative only: your run will score differently depending on the model and the persona.

On this transcript, task_completion for “Fubar” can come back Fail even though the agent sounded helpful: the transcript shows the agent saying “I’m glad I could help” and calling end_call right after the customer’s first “yeah, thanks”, without ever stating the password reset steps that outcome asks for. The cause is the instructions string in Step 2: it tells the agent to call end_call “when the customer confirms their issue is resolved,” but never requires it to state the reset steps first, so a quick acknowledgment from the simulated customer is enough to trigger end_call early.

Tightening that one instruction fixes it. Add a sentence to the instructions string in Step 2:

instructions=(
    "You are a helpful support agent. Be friendly and proactive. "
    "Ask clarifying questions and provide step-by-step guidance. "
    "Before calling `end_call`, you must have described the password "
    "reset steps out loud. "
    "When the customer confirms their issue is resolved, "
    "call the `end_call` tool to gracefully end the call."
),

Rerunning Step 2 and Step 4 with the tightened instructions: task_completion moves from Fail to Pass for the same persona, because the transcript now includes the reset steps before the agent hangs up. Illustrative only: your run will score differently depending on the model and the persona.