Prompt Workbench Simulation: Run Chat Simulations Without the SDK
Run multi-turn chat simulations against a saved prompt version directly from the Prompts workbench, no SDK or agent definition required.
Run multi-turn chat simulations against a saved prompt version straight from the Prompts workbench. The prompt acts as the agent: no SDK, no agent definition, no code. You’ll get CSAT and evaluation scores per conversation.
| Time | Difficulty | Package |
|---|---|---|
| 10 min | Beginner | Dashboard only |
- Future AGI account → app.futureagi.com
- At least one saved prompt version in the Prompts workbench (see Prompt Versioning if you need to create one)
- At least one chat scenario under Simulate → Scenarios (see Scenarios if you need to create one)
Install
No packages to install. This cookbook uses the Future AGI dashboard only. The Simulation tab runs entirely inside the Prompts workbench, using the prompt’s own system message, model, and parameters to drive the conversation.
Tutorial
Open your prompt in the workbench
Go to app.futureagi.com → Prompts (left sidebar under BUILD) → click the prompt template you want to test.
You should see the workbench open on the Playground tab, with Evaluation, Metrics, and Simulation as sibling tabs.
Open the Simulation tab
Click Simulation in the top tab bar.
Note
The tab is only clickable once the prompt has a saved version. If it shows “You need to submit at least one prompt before running simulations.”, go back to Playground and click Run Prompt to save v1.
You should see an empty Simulation tab with a Create Simulation button.
Create a simulation
Click Create Simulation. In the “Create Chat Simulation” dialog, fill in:
- Simulation Name: auto-populated as
Simulation - {Date} at {Time}, edit to something descriptive likesupport-prompt-v2-test - Prompt Version: the saved version to test, the default version is pre-selected
- Description (optional): what you’re testing, for example
Testing revised tone instructions against return-request scenario - Select Scenarios: check one or more scenarios, each checked scenario produces one simulated conversation per run
Tip
No scenarios yet? Click Create New Chat Scenario at the top of the list. It opens scenario creation in a new tab. Save, return to this dialog, and click the refresh icon to reload the list.
Click Create Simulation. This immediately starts the first run against your attached scenarios: the dialog closes, you’ll see a toast reading “Simulation created and execution started”, and the simulation detail view opens automatically with the executions grid filling in as conversations complete.
Review the header controls
The simulation detail view header has three controls: Version, Scenarios, and Evals.
Use Version to switch which prompt version the next run uses. The change saves to the simulation immediately. Use Scenarios to add or remove attached scenarios, the count badge shows how many are currently attached. Use Evals to open the evaluations drawer and click Add Evaluation to configure a metric that runs automatically on each future run.
Tip
Since execution starts as soon as you click Create Simulation, you can’t add evaluations before that first run. To score a run that already completed, select its row(s) in the executions grid, open Evals, and click Run Evaluation in the drawer footer. The drawer requires at least one row selected and prompts “Please select at least one Test Run to run Evaluation” otherwise. Task Completion, Tone, and the Conversational agent evaluation group give structured scores on top of raw CSAT.
You should see the Scenarios and Evals badge counts reflect what you attached.
Re-run the simulation
Click Run Simulation in the top-right corner of the simulation detail header to start another run, for example after switching the prompt version or attaching new scenarios.
You should see a success notification confirming execution has started, and a new row appear in the executions grid: it carries a Scenario column and a Total Chats count, and fills in as each conversation runs up to 10 turns between the prompt and the simulated persona.
Review results
Click any row in the executions grid to open the execution detail page at /dashboard/simulate/test/{simulationId}/{executionId}.
The execution detail page has three tabs: Chat Details, Analytics, and Optimization Runs.
Chat Details shows the full conversation transcript, every turn between the simulated persona and your prompt, along with aggregate metrics:
| Metric group | What it shows |
|---|---|
| Chat Details | Total chats, completed count, completion percentage |
| System Metrics | Avg total tokens, avg input tokens, avg output tokens, avg chat latency |
| Evaluation Metrics | Average score per configured evaluation |
Analytics shows evaluation score distribution charts across the run.
Back on the simulation detail view, each row in the executions grid represents one run. Open a row to read its chats and see the CSAT score.
You should see a CSAT score and, if you attached evals, colored evaluation tags on each completed row.
Iterate: swap versions and re-run
Say your first run against support-prompt-v1 returns a CSAT of 62 (illustrative) on the return-request scenario. Open the transcript on Chat Details and read where the persona lost patience: the prompt’s system message never tells the model to acknowledge the customer’s frustration before offering a solution, so the assistant jumps straight into policy details and the persona ends the chat unresolved.
Edit the prompt to add one instruction, for example “Acknowledge the customer’s issue in one sentence before proposing next steps,” and save it as a new version.
Use the Version dropdown to switch to the new prompt version, then click Run Simulation again.
You should see the new run append a row to the executions grid. All previous runs stay. In this example, CSAT rises from 62 to 81 (illustrative) on the same scenario, evidence that the acknowledgment instruction fixed what the persona reacted to.
You can now run multi-turn chat simulations against any prompt version, score them automatically, and compare runs across versions without writing any code.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Simulation tab won’t click, shows a tooltip | Prompt has no saved version yet | Go to Playground, click Run Prompt to save v1, then open Simulation |
| ”Select Scenarios” list is empty in the dialog | No chat scenarios exist for this project | Click Create New Chat Scenario, save it, then click the refresh icon in the dialog |
| Run Simulation is disabled with “Some selected scenarios have no datapoints. Remove them from the selection to run.” | A selected scenario has zero datapoints | Remove that scenario from the selection, or add datapoints to it, then try again |
| Run finishes but Evaluation Metrics column is blank | No evaluation was attached before the run | Select the row(s) in the executions grid, open Evals, add an evaluation, then click Run Evaluation in the drawer footer |
| Switching the Version dropdown doesn’t change scores | Version change alone doesn’t trigger a new run | Click Run Simulation again after switching versions |
| A row stays on Failed | The simulated conversation hit an error mid-run (model timeout or a malformed scenario turn) | Open the row’s transcript on the Chat Details tab to see where it stopped, fix the scenario, and re-run |
| Execution detail page is blank after clicking a row | Navigated to a stale execution URL after the simulation was deleted or recreated | Return to the Simulation tab and open a row from the current executions grid |
| CSAT looks unexpectedly low on an otherwise normal transcript | The simulated persona ended the conversation before reaching a resolution turn | Read the transcript on Chat Details. The persona’s exit condition is often the real cause, not the prompt |
Next: score simulations with a defined agent and persona set via the SDK in Chat Simulation with Personas.
Questions & Discussion