Chat Simulation & Fix My Agent

import os
import asyncio
import litellm
from fi.simulate import TestRunner, AgentInput
from fi.prompt import Prompt

FI_API_KEY = os.environ["FI_API_KEY"]
FI_SECRET_KEY = os.environ["FI_SECRET_KEY"]

print("Loaded FI_API_KEY and FI_SECRET_KEY" if FI_API_KEY and FI_SECRET_KEY else "Missing a key")

You should see Loaded FI_API_KEY and FI_SECRET_KEY. fi.simulate is the agent-simulate package’s import namespace; fi.prompt ships from futureagi.

PROMPT_NAME = "Customer_support_agent"
PROMPT_LABEL = "production"

prompt = Prompt.get_template_by_name(PROMPT_NAME, label=PROMPT_LABEL)
compiled = prompt.compile()

system_prompt = next(
    (m.get("content") for m in compiled if m.get("role") == "system"), ""
)
model_name = prompt.template.model_configuration.model_name

print(f"Loaded '{PROMPT_NAME}' ({PROMPT_LABEL}) on {model_name}")

You should see something like Loaded 'Customer_support_agent' (production) on gpt-4o-mini. compile() returns the same message list you’d hand to any LLM provider, so pulling the system message out of it keeps the simulation in sync with whatever you edit in the Prompt Workbench. Called with no kwargs, compile() leaves any {{variable}} placeholders unsubstituted, so those markers land in system_prompt verbatim. Use .get() rather than indexing: unresolved placeholder entries in compiled carry no role key, and chat prompts routinely have at least one.

def build_agent(system_prompt: str, model: str):
    async def agent_function(input_data: AgentInput) -> str:
        messages = []
        if system_prompt:
            messages.append({"role": "system", "content": system_prompt})
        messages.extend(input_data.messages)
        if input_data.new_message:
            messages.append(input_data.new_message)

        response = await litellm.acompletion(
            model=model,
            messages=messages,
            temperature=0.2,
        )
        return response.choices[0].message.content or ""

    return agent_function


agent_callback = build_agent(system_prompt, model_name)

test_input = AgentInput(messages=[], new_message={"role": "user", "content": "What's your refund policy?"})
test_reply = asyncio.run(agent_callback(test_input))
print(test_reply)

You should see a plausible reply about a refund policy printed to the console. agent_function is what TestRunner calls once per turn. AgentInput.messages carries the conversation so far and new_message carries the scenario’s next line; litellm routes model to whichever provider it belongs to. Confirming the reply here proves the LiteLLM model string works before you spend a full run finding out it doesn’t.

RUN_TEST_NAME = "Chat regression test"
CONCURRENCY = 5

async def main():
    runner = TestRunner(api_key=FI_API_KEY, secret_key=FI_SECRET_KEY)
    return await runner.run_test(
        run_test_name=RUN_TEST_NAME,
        agent_callback=agent_callback,
        concurrency=CONCURRENCY,
    )

report = asyncio.run(main())

You should see the SDK’s own console output as the run progresses: 🔍 Fetching Run Test ID for name: ..., then ✓ Test Execution Started: ..., and finally ✅ Cloud Simulation Completed.. run_test_name must match a run test you’ve already configured on the platform, and concurrency caps how many conversations run in parallel. The cloud path always returns report.results empty: results are read on the platform in the next step, not off the returned object.

Go to app.futureagi.com → Simulation → your run test name. You should see one row per conversation with the evaluation scores your run test configured (task completion, tone, groundedness, or whatever you attached).

On the refund-policy scenario, say task completion scores low (illustrative, not a score we actually observed here, but the kind of row that sends you to Fix My Agent).

Fix My Agent stays greyed out until the run has at least 15 connected calls, so a scenario set small enough to finish fast won’t unlock it.

From the run’s results page, click Fix My Agent. The panel opens empty the first time: it shows “There are no suggestions yet, click the refresh button to get suggestions”, so click refresh. Once populated, it reads every conversation against your evaluation criteria. It returns a ranked list of issues, each with a heading, a priority (High/Medium/Low), a written recommendation, and a Calls Affected count.

For the refund-policy failure, an entry might read: “Agent gives inconsistent refund windows” (High priority, 6 calls affected), recommending the system prompt state the refund window as a fixed number of days instead of leaving it to the model’s judgment. There’s no prompt text to copy here. Instead, click Optimize My Agent, which opens optimization setup scoped to this run and its findings.