Queues & Items

A queue is the campaign, an item is what’s inside it

A queue is the campaign. It carries:

• a status
• the labels it’s collecting answers for
• the people working it and the role each one holds
• submissions per item: how many independent annotators must complete an item before it counts as done
• whether review is required before a submission counts

An item is one thing inside that campaign, pulled from one of the sources Annotation covers (see Understanding Annotation). It lands in the queue when someone adds it there (see Add items), moving from untouched to answered as work happens.

A queue moves through four statuses of its own:

• draft: doesn’t accept annotations yet
• active: the only status that accepts submissions, flips on the moment someone activates it
• paused: temporarily closed to new submissions without marking the queue done
• completed: closed to further submissions, either because a manager marked it done directly or because every item finished on its own; a skipped item blocks that automatic path, so a manager has to close the queue by hand instead

How a queue and its items relate

An item carries a status of its own, plus a review state layered on top of it when the queue requires review.

The status moves through four values:

• pending: waiting to be picked up
• in_progress: being worked on
• completed: done
• skipped: passed over without an answer

On a queue where submissions per item is 1, opening an item reserves it for whoever opened it, whether it’s still pending or already in_progress, so a second annotator can’t pick up the same one while it’s being worked. If the queue’s reservation timeout passes before they act on it, the reservation simply lapses and the item becomes claimable again, the status itself doesn’t move. On a queue that needs more than one submission per item, that reservation doesn’t apply, so more than one annotator can open the same item at once. See Queue settings & limits for the exact timeout options.

An item is done once the required number of annotators have each completed it, with every label the queue requires scored, unless the queue calls for review. When review is required, a submitted item’s review state moves to pending review instead of the item completing outright. A reviewer’s approval then completes it, and sending it back moves the status to in_progress so the annotator can act on the feedback.

flowchart TD
accTitle: How a queue and its items relate
accDescr: A queue carries a status, its attached labels, and its annotators with roles, and it holds items. Each item moves through its own status and completes only once every label the queue requires has enough submissions, routing to pending review first when the queue requires it. From pending review, a reviewer's approval completes the item, and sending it back returns it to in_progress.
Q["Queue"] -->|"has"| QL["Labels attached to the queue"]
Q -->|"has"| QA["Annotators: roles annotator, reviewer, manager"]
Q -->|"holds"| IT["Item: pending"]
IT --> INP["Item: in_progress, reserved"]
INP --> D{"Every label the queue requires scored by enough annotators?"}
D -->|"no"| INP
D -->|"yes, review off"| DONE["Item: completed"]
D -->|"yes, review on"| REV["Item: pending review"]
REV -->|"approve"| DONE
REV -->|"send back"| INP

Roles decide who can do what

Everyone on a queue holds one or more of three roles. An annotator submits scores. A reviewer approves or sends back submissions when the queue requires it. A manager configures the queue and its people. Only annotators and managers can actually submit an annotation: holding the reviewer role by itself doesn’t grant that.

Two shortcuts save you from adding people by hand. Whoever creates a queue is added to it as a manager the moment it’s saved, and admins act as managers automatically without ever being added as members: org admins on every queue in their org, workspace admins on every queue in their workspace.

The default queue you meet before you build one

You’ll often run into a queue before you ever set one up yourself. Future AGI creates at most one default queue per project, dataset, or agent definition, and only the first time that scope actually needs one, not automatically for every one of them, so a project nobody has annotated yet has no default queue. Unlike an ordinary queue, it never auto-completes: because it’s meant to keep collecting whatever lands in it indefinitely, no amount of finished items closes it on its own.

Why it matters

The queue is what makes annotation a managed workflow instead of something one person does off to the side. The item is what keeps that workflow honest one row at a time, holding its own status so a single stuck item never quietly skews the read on how the whole batch is progressing. It’s also why the same submissions per item rule works whether that count is one or five: the queue sets the bar, and every item is measured against it independently.

Keep exploring