Scores
Understand the Score model -- the unified annotation primitive that stores labels, values, and metadata across all source types.
What it is
A score is the atomic data record created every time an annotation label is applied to a source entity. It is the unified annotation primitive in FutureAGI, replacing the legacy TraceAnnotation model with a single structure that works identically across traces, spans, sessions, dataset rows, prototype runs, and simulation executions.
Every score answers five questions: what was annotated (source), how it was annotated (label and value), who annotated it (annotator), when (timestamps), and why (optional notes and queue context).
Score fields
| Field | Type | Description |
|---|---|---|
id | UUID | Unique identifier for the score. |
label_id | UUID | The annotation label that was used. Determines the expected value format. |
value | JSON | The annotation value. Format varies by label type (string, number, boolean, string array). |
source_type | string | What kind of entity was annotated. One of the six supported source types. |
source_id | UUID | The ID of the annotated entity (e.g. trace ID, dataset row ID). |
annotator | string | Who created the annotation — a user email or system identifier. |
score_source | string | Origin of the score: human (manual annotation), model (LLM-generated), or auto (rule-based). |
notes | string | Optional free-text notes attached to the annotation. Available when Allow Notes is enabled on the label. |
queue_item | UUID | Optional. Links the score to a specific queue item if it was created through the queue workflow. Null for inline annotations. |
created_at | datetime | When the score was created. |
updated_at | datetime | When the score was last modified. |
Source types
Scores can target any of the following entity types. The source_type and source_id fields together form a polymorphic foreign key to the annotated entity.
| Source Type | Entity | Where it appears |
|---|---|---|
trace | An LLM trace from Observe | Trace detail view, LLM Tracing grid |
observation_span | A specific span within a trace | Span detail within trace tree |
trace_session | A conversation session (group of traces) | Sessions grid and session detail |
dataset_row | A row in a dataset | Dataset table view |
call_execution | A simulation call execution | Simulation results view |
prototype_run | A prototype/experiment run | Prototype results view |
Two ways to create scores
Queue workflow (managed)
Scores created through an annotation queue are linked to a queue item via the queue_item field. The queue manages assignment, progress tracking, and completion logic.
Add items to a queue
Select entities (traces, dataset rows, etc.) in their respective views and click Add to Queue. Each becomes a queue item.
Annotate through the workspace
Click Start Annotating on the queue detail page. The workspace presents items one at a time with the queue’s labels. Each submitted annotation creates a score.
Auto-completion
When all labels are scored by the required number of annotators, the queue item auto-completes.
Inline annotation (direct)
Scores can also be created directly from the detail view of any supported entity — without going through a queue. Inline annotations have queue_item set to null.
- Trace detail: Open a trace, expand the annotation panel, and apply any label from your organization.
- Session grid: Annotation columns appear directly in the sessions table for quick scoring.
- Dataset view: Annotate individual rows from the dataset table.
Inline annotations are ideal for ad-hoc feedback during investigation or review. They produce the same Score records and appear alongside queue-created scores in all views and exports.
Value formats by label type
The value field in a score is JSON. Its shape depends on the label type:
| Label Type | Value Example | JSON Type |
|---|---|---|
| Categorical (single) | "Positive" | string |
| Categorical (multi) | ["Relevant", "Accurate"] | string array |
| Numeric | 7 | number |
| Text | "Consider rephrasing the second paragraph." | string |
| Star Rating | 4 | number |
| Thumbs Up/Down | true | boolean |
Where scores appear
Scores are surfaced everywhere the annotated entity is displayed:
- Trace detail view — Annotation panel shows all scores for the trace and its spans.
- Sessions grid — Dynamic annotation columns display score values inline with session data. Filter and sort by annotation values.
- Dataset table — Score values appear as columns alongside dataset row data.
- Queue detail — The items tab shows all scores submitted for each queue item.
- API — Query scores programmatically with filters on source type, label, annotator, and date range.
Bidirectional sync
Scores created through different paths stay synchronized:
- An annotation submitted on a trace via Observe automatically creates a corresponding score visible in any queue containing that trace.
- A score submitted through a queue workflow is immediately visible in the trace detail and session grid views.
This ensures a single source of truth regardless of where the annotation originated.