Understanding Annotation

Label, queue, item, score

Annotation is how a person turns their judgement on a piece of AI output (a rating, a category, a correction) into a record Future AGI can compare against every other judgement made the same way. An annotator, someone on your team, works through a queue, or judges a source directly. Either path ends at the same four objects: a label, a queue, an item, and a score.

A label is a reusable question with a fixed answer type: text, a number, a category, a star rating, thumbs up or down. Define it once and reuse it wherever you want that same question answered.

A queue attaches one or more labels and holds the items waiting to be judged against them. A queue can’t exist with zero labels; the labels are what an annotator sees when they open an item.

Each item a queue holds points at exactly one source:

• A trace
• A span
• A session
• A Simulation (a simulated voice or text call)
• A prototype run (a prompt run over a dataset)
• A dataset row

There’s no such thing as an item pointing at two sources, or none.

Every answer to a label, whether it came from working an item or from judging a source directly, becomes one score.

How the four pieces fit

flowchart TD
accTitle: The Annotation object model
accDescr: A label attaches to a queue, and a queue holds items. An item points at exactly one of six source kinds, and so does every score.
L["Label · fixed answer type"] -->|attached to| Q["Queue · one or more labels"]
Q -->|holds| I["Item"]
SRC{{"Source · exactly one of:<br/>trace, span, session,<br/>Simulation, prototype run,<br/>dataset row"}}
I -->|is about| SRC
I -->|produces| SC["Score"]
SC -->|points at| SRC
L -->|shapes the answer on| SC

One trace, judged two ways

Take a trace from the support-agent project. Route it into the Support quality review queue, and it becomes an item whose one source is that trace. An annotator opens the item, answers the queue’s Response quality label, and that answer lands as a score referencing the trace, the label, and the item.

The same trace is judged directly with no queue picked, in the trace’s own view in Observe, and someone answers Response quality on it there. That judgement doesn’t skip the item: the inline judgement lands as the same kind of score record, pointing at the trace and the label. Both show up wherever the trace’s judgements are read back.

Why it matters

• An item points at exactly one source, so anything reading the score back never has to guess which of six possible sources it was about
• A judgement made inline is never a lesser record: it resolves to a queue item just like a queue-worked one, so filtering, exporting, or displaying scores never has to special-case where they came from
• A label defined once and attached wherever it’s needed means the same question, Response quality in a support queue and in a compliance queue, produces answers that land in one comparable set of scores

Keep exploring