Simulation FAQ & fixes

In this page

The questions people ask most about Simulation, and the errors they run into, with a direct fix for each. Hit an error? Jump straight to Common errors and fixes. If your answer isn’t here, reach out via support.

Common errors and fixes

Symptom	Cause	Fix
Scenario generation comes back Failed	An uploaded script or SOP is a scanned PDF with no text layer, or an imported dataset is under 10 rows, has duplicate column names, or has a persona column typed as plain text	Fix the source material: scripts and SOPs need a real text layer, datasets need at least 10 rows, unique column names, and a persona column typed as Persona if one exists
A scenario is greyed out in the run wizard, tooltip “This scenario has no datapoints to run against”	The scenario has 0 rows	Open it and add rows before selecting it in a run
”No.of rows” or “No. of scenarios” rejects the number you typed	Row count has to be between 10 and 20,000	Enter a value in that range
The Add Column drawer won’t save more than 10 fields in one pass	Ten columns is the limit per save	Save those 10, then reopen Add Column for more; there’s no cap on the scenario as a whole
Re-run simulation is disabled, or missing entirely	The run has zero completed calls yet, or it’s simulated from a Prompt Workbench prompt, which never gets a rerun	Wait for at least one call to finish; a prompt-sourced run needs a new simulation instead
Removing an eval from a run is blocked	It’s the last eval left on that run	Add a replacement eval first; a run always needs at least one
Fix My Agent stays disabled	The run isn’t complete yet, or it has fewer than 15 connected calls	Wait for the run to finish with 15 or more connected calls; the tooltip names which condition is missing
Tool call evaluation is on but nothing gets scored	The agent is on Retell or Bland.ai, and tool call evaluation only works for Vapi	Switch the agent definition to Vapi if tool calls need scoring
A chat run’s SDK script finishes, but the run stays empty on the dashboard	run_test_name didn’t match the run’s name exactly, or FI_API_KEY/FI_SECRET_KEY are missing or wrong	Copy the name from the dashboard’s boilerplate panel instead of retyping it, and confirm both keys are set
report.results is always an empty list after run_test	Cloud mode doesn’t populate that field; your results live on the dashboard, not in the SDK’s return value	Read outcomes from the run’s Simulated runs, Call Details (Chat Details on a chat run), and Analytics tabs, not the object run_test returns
A call fails with a generic error instead of the exception your callback raised	The SDK reports a generic error to the dashboard rather than forwarding your exception’s message	Log the exception yourself inside the callback if you need to know what actually went wrong

Getting started

What do I need before I can run a simulation?

An agent definition with at least one version, and a scenario built against it with at least one eval attached. Connect your agent and Create scenarios cover both.

Do I need a deployed agent to run anything?

Not for chat. Simulate a prompt runs a saved Prompt Workbench version directly, with no agent definition and nothing to connect.

Do I need to write code?

Not for voice: Future AGI places the calls itself. A chat run needs a short Python callback using the agent-simulate SDK, which Run a chat simulation walks through.

Which voice providers are supported?

Vapi, Retell, and Bland.ai. See Voice providers for what each one needs and supports, and use Others for an agent you can reach by phone.

Scenarios and generation

Why is a scenario greyed out when I try to select it for a run?

It has 0 rows. Open it and add rows before it can be picked.

What’s the smallest and largest scenario I can generate?

Between 10 and 20,000 rows, whether you’re generating a new scenario or adding rows to an existing one.

How many columns can I add at once?

Up to 10 in a single save from the Add Column drawer. Reopen it for more; there’s no ceiling on the scenario itself.

Why did my dataset import get rejected?

A dataset needs at least 10 rows, no duplicate column names, and a persona column, if it has one, typed as Persona rather than plain text. The rejection names which condition failed.

Running a simulation

My chat run’s Simulated runs tab is still empty. Is it broken?

No. A chat run stays empty until you run the SDK script yourself: it isn’t queued anywhere and it doesn’t time out. An empty tab means the script hasn’t run, not that the run failed.

Why did my voice call cut off partway through?

Every call is capped at 30 minutes and ends automatically at that mark, regardless of provider. See Call metrics for how a call’s duration is reported once it completes.

Can I switch a run to a different voice provider after creating it?

No. The provider lives on the agent definition, not on the version, and both are fixed once a run test is created. Point a new run at a different agent definition instead. Run a voice simulation covers why the version matters more for voice than for chat.

Why does my chat conversation stop after 50 turns even though it shouldn’t have ended yet?

50 turns is a safety cap that applies when a scenario’s end condition never triggers. If conversations are cutting off early, check the scenario’s flow rather than the agent.

Reruns and replay

Why does my chat run only offer “Run Evals”, never “Run test + Evals”?

A chat agent’s calls live in your own code, so the platform has nothing to replay; only its evals can rerun. Voice runs get both options, covered in Run a voice simulation.

If a rerun fails right after I click it, did I lose the call’s original data?

For Run Evals, no: a rerun that fails before it starts leaves the call exactly as it was. Run test + Evals is different: it clears the call’s recording, transcript, and cost data as soon as it’s dispatched, before the new call is placed, so a failure right after that point doesn’t leave the original untouched. Edit evals in a simulation covers what Run Evals changes and what it leaves untouched.

Does rerunning overwrite my previous result?

No. The prior state is kept so you can compare before and after. See Runs & results.

Why can’t I replay this voice call’s exact configuration?

Configuration replay only reconstructs Vapi calls. A Retell or Bland.ai call still replays, but you get a transcript comparison back rather than the original provider setup. Replay voice calls has the detail.

Evals and tool calls

Why can’t I delete this eval?

It’s the last one on the run, and a run always needs at least one. Add a replacement before removing it.

I turned on tool call evaluation, but nothing got scored.

Two possible reasons: the agent is on Retell, where tool call evaluation isn’t wired at all, or the scenario simply never reached a tool call. Evaluate tool calls covers both.

Why don’t tool calls show up in the transcript?

They’re intentionally excluded from the transcript view. Check that call’s tool-call results instead, alongside its other eval scores, covered in Calls & transcripts.

My chat agent calls tools, but tool call evaluation still finds nothing to score.

The callback has to return an AgentResponse with tool_calls and tool_responses set, not a plain string. Evaluate tool calls has the exact shape.

Fix My Agent and optimization

Why is the Fix My Agent button disabled?

Two conditions, and the tooltip names which one is missing: the run has to be complete, and it needs at least 15 connected calls behind it. See Fix My Agent.

Fix My Agent says there are no suggestions. Is that an error?

No. Click refresh to run the analysis; if it genuinely finds nothing worth flagging, it says so rather than inventing an issue.

Does applying an optimization update my agent automatically?

No. An optimization run hands back a ranked list of trials and a best-performing prompt, but nothing pushes that prompt onto your agent for you. Copy the winning trial’s text into a new agent version yourself, covered in Optimization runs.

SDK

Which keys does the SDK need?

FI_API_KEY and FI_SECRET_KEY, read from your environment by TestRunner(), or passed directly as api_key/secret_key.

My keys look right, but the run stays empty. What’s going on?

Missing credentials don’t fail immediately, they only log a warning, then fail once the SDK actually calls the backend. Check both keys, and confirm run_test_name matches the run’s name exactly.

Why does report.results come back empty even though the run completed?

That field isn’t populated in cloud mode. Read outcomes from the run’s Simulated runs, Call Details (Chat Details on a chat run), and Analytics tabs on the dashboard instead of the object run_test returns.

My callback raised an exception. Why does the dashboard just show a generic error?

The SDK reports a generic failure to the dashboard rather than your exception’s message. Log it yourself inside the callback to see the real cause.

Can I return raw tool output without building tool_calls/tool_responses by hand?

Yes. Pass it through metadata={"tool_outputs": [...]} on your AgentResponse and the SDK converts it. Evaluate tool calls shows both paths.

Keep exploring