Run an experiment
Set up an experiment, run it across your dataset, and use Choose winner to pick the best configuration.
An experiment runs every prompt or agent and model combination you set up against the same dataset, scored by the evals you attach, so you can compare configurations side by side instead of testing them one at a time.
The experiment grid
One experiment lays your dataset’s rows against every configuration you add. A configuration pairs one prompt or agent with one model, so attaching three models to the same prompt gives you three configurations, one column each. Every eval you attach scores every configuration against the same rows, which is what makes the columns comparable.
flowchart TD
accTitle: The experiment grid
accDescr: Dataset rows and configurations cross to form the experiment grid. Each configuration pairs a prompt or agent with a model. Evals attach to the grid and score every configuration.
DS["Dataset rows"] --> GRID{{"Experiment grid"}}
PA1["Prompt or agent"] --> CFG1["Configuration A"]
MD1["Model"] --> CFG1
PA2["Prompt or agent"] --> CFG2["Configuration B"]
MD2["Model"] --> CFG2
CFG1 --> GRID
CFG2 --> GRID
EV["Evals"] -->|"scores every column"| GRID
Build the experiment
Click Experiment on the dataset to open the creation flow. It’s a three-step form.
Basic Info
Name the experiment and choose its type: LLM, TTS, STT, or Image. The type decides the output format and which models you can attach.
Configuration
Add the prompts or agents you want to compare and attach a model to each. Every prompt/agent-model pair becomes its own configuration column. LLM experiments can mix prompts and agents in the same run and attach tools to a prompt, useful for deciding whether an agent earns its extra complexity over a plain prompt; TTS, STT, and Image experiments take prompts only.
Evaluations
Optionally pick a column to compare outputs against as a baseline, then add the evals that will score every configuration.
Click Run Experiment, and every row runs against every configuration, with each output scored by your evals as it comes in.
Note
An eval you add after the run sits queued for a few seconds before it starts scoring.
Stop and rerun
Each experiment stops and reruns independently. Stop a running one without touching the others in the dataset, and rerun a completed, failed, or cancelled one later without setting it up again, though rerunning overwrites its existing results. Select more than one experiment at a time to rerun or delete them together.
Choose a winner
The experiment summary already lists every configuration. Once every configuration has a score, click Choose winner to open Winner Settings, where you set the importance of Average Response Time, Completion tokens, Total tokens, and each eval. Click Save & Run and the summary marks the winning configuration.
Tips
- Failed-only rerun: when transient failures (rate limits, network blips) leave a few cells errored, use the failed-only rerun on the column to recover them without redoing successful rows
Dive deeper
Questions & Discussion