Running Evaluations

Evaluations have no separate quickstart. An eval scores data, so you run it wherever your data already lives: a dataset, your live traces, a simulation, the playground, the SDK, or a CI job. The template you run is the same on every surface. The surface only changes what you feed it and where the score lands.

This page is the entry point. Pick your surface below and follow its guide.

Where evals run

%%{init: {"flowchart": {"curve": "linear"}}}%%
flowchart LR
  T["One eval template"] --> D["Dataset"]
  T --> TR["Traces"]
  T --> S["Simulation"]
  T --> P["Playground"]
  T --> K["SDK"]
  T --> C["CI/CD"]
  D --> SC["Comparable scores"]
  TR --> SC
  S --> SC
  P --> SC
  K --> SC
  C --> SC

In evaluation, offline and online describe the data, not your connection: an offline run scores data you’ve already stored (dataset rows, experiment outputs), an online run scores live production traffic as it arrives.

Surface	Where you run it	Guide
Dataset and experiments	Offline, over every row of a dataset	Run experiments
Traces	Online, on live spans, traces, and sessions	Set up evals in Observe
Simulation	Over simulated conversations	Run a simulation
SDK	Programmatic runs, with local Code Evals that need no API key	Evaluation SDK
CI/CD	On every pull request, gating the merge on eval scores	Evaluate in CI/CD

The online how-to for scoring live traces lives in Observe; this hub only points to it. Everything else runs from its own module, with the same templates and the same scores.

Dive deeper