Setup evals

Attach evaluations to your traces so every response is scored on the metrics you care about.

Evals score the responses your app produces, so you measure quality, safety, or accuracy on real production traffic instead of eyeballing traces one by one. This guide sets up an eval on the support-agent project from the Observing a LangGraph agent cookbook, using a toxicity eval as the worked example.

Open the eval builder

In the project’s trace explorer, click Add Evals at the top right.

Observe trace table for the support-agent project with the Add Evals button highlighted at the top right

Start from Add Evals in the project’s trace explorer

Name the task and pick a level

Evals run as a task. Give it a name and confirm the target project. Under Run evaluations on, choose the level the eval scores: Spans (one step), Traces (a whole request), or Sessions (a whole conversation).

Create Task page with a task name, the support-agent project, and Spans, Traces, and Sessions level options

Name the task, target the project, and pick the level to score

Add an evaluation

Under Evaluations, click Add Evaluation to open the library.

Create Task page with the Add Evaluation button highlighted and no evaluations added yet

Open the evaluation library from Add Evaluation

The library holds built-in templates grouped by category (Safety, RAG, Hallucination, Conversation, and more), each with its output type: Pass/fail, Percentage, or Choices. Search or filter, then click Add on the one you want. This example uses the built-in toxicity eval.

Select Evaluation library listing built-in templates like toxicity and conversation_hallucination with category filters and output types

Pick a template; here the built-in toxicity eval

Configure the eval

The name autofills. For a built-in eval the instructions and output type are pre-configured and read-only. Two things to set:

  • Error localization (optional): tick it to pinpoint which part of the input caused a failure
  • Variable mapping (required): map each of the eval’s inputs to a column on your traces. The panel on the right lists the columns available, so map the eval’s output to the trace’s output

Click Add Evaluation to attach it to the task.

Toxicity eval configuration showing the read-only instructions, Pass/fail output, an error-localization checkbox, and a variable-mapping panel

Map the eval’s variables to your trace columns, then add it

Schedule and run

The eval now appears under Evaluations. Set when it runs:

  • Historical data: score existing traces from a past time window (pick the window, for example 7D)
  • New incoming data: score every new trace as it arrives

Then tune Row limit and Sampling rate, the share of matching rows actually scored, where a lower rate runs faster and cheaper. Click Test to dry-run against a sample, or Create Task to start.

Create Task page with the added toxicity eval, Historical vs New incoming options, time window, row limit, and sampling rate, plus Test and Create Task buttons

Choose Historical or New incoming, tune sampling, then Create Task

Read the results

Scores land as a new column in the trace table, one result per row. Open any trace for the per-span score, and reuse the eval in filters (scores.<name>) and alerts to catch regressions before a customer does.

Trace table with an added eval column showing a Pass or Fail result on each support-agent row

The eval shows up as a column in the trace table, scored per row

Was this page helpful?

Questions & Discussion