Overview

What is Annotation?

Annotation captures human judgement on AI output and stores every judgement as a score you can filter, export, and turn into a dataset. The judged thing can be a trace, a span, a session, a call execution, a prototype run, or a dataset row.

Labels, queues, and scores

Three objects carry the whole model:

• A label is the question you ask: a reusable definition of what you’re judging, with a fixed answer type. Response quality, for instance, is categorical with three options: Good, Needs work, Wrong
• A queue organises who answers it and on what: a managed campaign that assigns items, the individual pieces of output being judged, to annotators and tracks their progress. Attach Response quality to a Support quality review queue and every annotator working it answers that same question
• A score is the answer itself, one record per judgement. An annotator answering Response quality on an item in Support quality review produces one score: Good

You can produce a score two ways: work an item through a queue, or annotate it inline, on the spot, with no queue involved.

Start here

Concepts