Evaluate prompt outputs

Read scores alongside outputs, compare them across versions, and remove an eval when you're done with it

Attach an eval template to a prompt template and its score shows up right next to the output it scored, inside the same editor you ran the prompt in. This guide continues with support-agent, the same template you ran in Run a prompt, and scores the output that run produced. See Evaluations for what an eval checks and how it arrives at that score; this page only covers wiring one to a prompt.

Run the prompt first

The Evaluation tab stays disabled until the template has at least one output to score. Open it too early and the tab shows why, in its own tooltip: “You need to submit at least one prompt and get an output before accessing the evaluation.” Run support-agent once and the tab unlocks.

Attach an eval

Open Evaluation

Open the Evaluation tab and click Add Evaluations.

Pick an eval template

Choose one from the list, for example customer_agent_human_escalation.

Note

If support-agent doesn’t have what the eval needs, pick a different template whose required inputs actually match what support-agent produces.

Map its required inputs and name the eval

Point each required input at one of support-agent’s own variables (company_name, customer_question), at model_input, or at model_output, and give the eval a name.

Fill in every required input

Leave one required input unmapped and Save Eval blocks with “Required input mappings must be filled” until every required input has a target.

Save the eval

Click Save Eval. The eval attaches to support-agent and scores the output you already have.

Read the results next to each output

Each attached eval’s score lands in its own column, next to the output it scored, so you can scan outputs and scores together instead of cross-referencing two views.

The table starts wide: Show Variables is on by default, so each variable’s value already shows up as its own column, useful when a low score comes from what the model was actually given rather than the model itself. Turn it off to narrow the table down to outputs and scores. Show Prompts is off by default; turn it on to add a header band above the output columns showing the prompt messages.

Run the same evals across several versions

You don’t have to reattach an eval to every version by hand. Once the prompt template has more than one committed version, click the + in the comparison column header of the results table. In the Add version to compare drawer that opens, tick one or two more versions to bring into the table; the current version is already ticked and locked. Click Compare and the table now shows outputs and scores side by side for every version you picked.

With several versions in view, reopen Add Evaluations: it lists every eval already attached to the template with a checkbox next to each. With nothing checked, the button reads Run All and runs every attached eval; check specific ones and it switches to Run Selected, running just those. Future AGI runs the evals against every version now in the table, in one pass. Two things can trip this: the template needs at least one eval attached before there’s anything to run, and an eval that belongs to a different template than the one you’re comparing gets rejected.

Remove an eval

Open Add Evaluations and click the trash icon on the eval’s row. Confirm “Delete this evaluation and its results?”; this can’t be undone. Removing it drops the eval off the list, and its column and scores disappear from the results table.

Questions & Discussion