Queues

About

An annotation queue is a managed campaign that groups items to annotate, assigns them to annotators, tracks progress, and enforces quality controls. Queues sit between labels (what to measure) and scores (the resulting data), providing the operational layer that turns annotation from an ad-hoc activity into a structured workflow.

---

Creating a Queue

---

Assignment Strategies

Strategy	Behavior	Best For
Manual	Annotators browse and pick items themselves from the queue list.	Small queues or exploratory annotation where annotators need context to choose.
Round Robin	Items are distributed cyclically across annotators in rotation.	Even distribution when annotators work at similar speeds.
Load Balanced	Items are distributed based on each annotator’s current workload.	Teams with varying availability or part-time annotators.

---

Multi-Annotator Support

For tasks that benefit from agreement between multiple reviewers, set the Annotations Required field (1-10).

• Each item must receive the configured number of complete annotations before it transitions to Completed.
• Different annotators independently annotate the same item. They do not see each other’s responses.
• The queue analytics tab shows inter-annotator agreement metrics once multiple annotators have scored the same items.

---

Reservation System

When an annotator opens an item, the system reserves it for a configurable timeout period. This prevents two annotators from working on the same item simultaneously.

• Default timeout: 30 minutes
• Configurable range: 15 minutes to 4 hours
• Expiry behavior: If the annotator does not submit or skip within the timeout, the reservation expires and the item returns to Pending for another annotator

---

Review Workflow

Enable Requires Review on a queue to add a review step after annotation:

1. Annotators complete their work as usual. When all required annotations are submitted, the item moves to Pending Review instead of Completed.
2. A designated reviewer opens the item, sees all submitted annotations, and either Approves (moves to Completed) or Rejects (sends back to Pending for re-annotation).

This is useful for high-stakes labeling tasks where a senior reviewer must validate annotations before they become final.

---

Queue Lifecycle

Status	Description	Can transition to
Draft	Queue is being set up, not yet accepting annotations	Active
Active	Annotators can annotate items	Paused, Completed
Paused	Temporarily stopped, no new annotations allowed	Active, Completed
Completed	All items done or manually completed	Active (re-open)

Activating a queue

A newly created queue starts in Draft. To begin accepting annotations, use the menu and select Activate, or open the queue detail page and change the status.

Auto-completion

Items auto-complete when:

1. All labels attached to the queue have been scored for the item
2. The required number of annotators have each fully annotated the item
3. If Requires Review is enabled, the reviewer has approved the item

---

Item Statuses

Status	Meaning
Pending	Waiting for an annotator to pick it up
In Progress	An annotator has opened the item and is actively annotating
Completed	All required annotations have been submitted
Skipped	An annotator chose to skip this item. It remains available for others.
Pending Review	Annotations are done but awaiting reviewer approval (when review workflow is enabled)

---

Managing Queues

Action	How
Edit	Open the queue detail page, use the Settings tab to modify name, labels, annotators, or workflow settings
Duplicate	Use the menu and select Duplicate. Creates a copy in Draft status.
Archive	Use the menu and select Archive. Soft-deletes the queue.
Search and filter	Use the search bar to filter by name and the status dropdown to filter by queue status

---

Next Steps