What it is
A score is the atomic data record created every time an annotation label is applied to a source entity. It is the unified annotation primitive in FutureAGI, replacing the legacy TraceAnnotation model with a single structure that works identically across traces, spans, sessions, dataset rows, prototype runs, and simulation executions. Every score answers five questions: what was annotated (source), how it was annotated (label and value), who annotated it (annotator), when (timestamps), and why (optional notes and queue context).Score fields
| Field | Type | Description |
|---|---|---|
id | UUID | Unique identifier for the score. |
label_id | UUID | The annotation label that was used. Determines the expected value format. |
value | JSON | The annotation value. Format varies by label type (string, number, boolean, string array). |
source_type | string | What kind of entity was annotated. One of the six supported source types. |
source_id | UUID | The ID of the annotated entity (e.g. trace ID, dataset row ID). |
annotator | string | Who created the annotation — a user email or system identifier. |
score_source | string | Origin of the score: human (manual annotation), model (LLM-generated), or auto (rule-based). |
notes | string | Optional free-text notes attached to the annotation. Available when Allow Notes is enabled on the label. |
queue_item | UUID | Optional. Links the score to a specific queue item if it was created through the queue workflow. Null for inline annotations. |
created_at | datetime | When the score was created. |
updated_at | datetime | When the score was last modified. |
Source types
Scores can target any of the following entity types. Thesource_type and source_id fields together form a polymorphic foreign key to the annotated entity.
| Source Type | Entity | Where it appears |
|---|---|---|
trace | An LLM trace from Observe | Trace detail view, LLM Tracing grid |
observation_span | A specific span within a trace | Span detail within trace tree |
trace_session | A conversation session (group of traces) | Sessions grid and session detail |
dataset_row | A row in a dataset | Dataset table view |
call_execution | A simulation call execution | Simulation results view |
prototype_run | A prototype/experiment run | Prototype results view |
Two ways to create scores
Queue workflow (managed)
Scores created through an annotation queue are linked to a queue item via thequeue_item field. The queue manages assignment, progress tracking, and completion logic.
Add items to a queue
Select entities (traces, dataset rows, etc.) in their respective views and click Add to Queue. Each becomes a queue item.
Annotate through the workspace
Click Start Annotating on the queue detail page. The workspace presents items one at a time with the queue’s labels. Each submitted annotation creates a score.
Inline annotation (direct)
Scores can also be created directly from the detail view of any supported entity — without going through a queue. Inline annotations havequeue_item set to null.
- Trace detail: Open a trace, expand the annotation panel, and apply any label from your organization.
- Session grid: Annotation columns appear directly in the sessions table for quick scoring.
- Dataset view: Annotate individual rows from the dataset table.
Value formats by label type
Thevalue field in a score is JSON. Its shape depends on the label type:
| Label Type | Value Example | JSON Type |
|---|---|---|
| Categorical (single) | "Positive" | string |
| Categorical (multi) | ["Relevant", "Accurate"] | string array |
| Numeric | 7 | number |
| Text | "Consider rephrasing the second paragraph." | string |
| Star Rating | 4 | number |
| Thumbs Up/Down | true | boolean |
Where scores appear
Scores are surfaced everywhere the annotated entity is displayed:- Trace detail view — Annotation panel shows all scores for the trace and its spans.
- Sessions grid — Dynamic annotation columns display score values inline with session data. Filter and sort by annotation values.
- Dataset table — Score values appear as columns alongside dataset row data.
- Queue detail — The items tab shows all scores submitted for each queue item.
- API — Query scores programmatically with filters on source type, label, annotator, and date range.
Bidirectional sync
Scores created through different paths stay synchronized:- An annotation submitted on a trace via Observe automatically creates a corresponding score visible in any queue containing that trace.
- A score submitted through a queue workflow is immediately visible in the trace detail and session grid views.
What you can do next
Annotation Labels
Learn about the five label types that define score value formats.
Queues & Workflow
Understand how queues manage the annotation lifecycle and produce scores.
Quickstart
Walk through the full flow from label creation to submitted scores.