Create a queue

A queue attaches one or more labels to a set of items, adds the people who’ll answer them, and sets how many independent submissions each item needs. This walks through the create-queue drawer in the order it presents its fields, building one running example: Support quality review, carrying the Response quality label with two submissions per item.

Open the drawer

Go to Annotations in the sidebar, switch to the Queues tab, and click Create Queue to open the drawer. A queue can belong to a project, a dataset, or an agent definition, or to none of them at all, an org-level queue. This drawer doesn’t ask you to pick one, so Support quality review comes out org-level.

Name it and describe it

Queue Name is required and free text: type Support quality review. The name has to be unique, case-insensitively, among the queues you haven’t archived in that same scope, so a name already used there is rejected.

Description is optional, a line or two on the queue’s purpose.

Attach labels

Pick the labels annotators will answer for every item in this queue. At least one is required. The drawer won’t let you save without it. Attach Response quality.

Add annotators

Add the workspace members who’ll work this queue as annotators. The picker opens with your own row already selected as Annotator, Reviewer, and Manager, tagged (creator), so you already count toward submissions per item. Untick Annotator on your row if you don’t want to answer items yourself. For Support quality review, add one more member: you and them make two.

Auto-assign items to all annotators is a checkbox alongside the picker. Turn it on and every annotator is assigned to every item, so anyone can open anything. Leave it off and assignment is manual instead, which means someone has to hand items out (covered in Add items).

Set submissions per item

Submissions per item is how many different annotators have to answer each item before it’s done. It can’t exceed the number of annotators you’ve added. Set it to 2 for Support quality review, so every item needs two independent takes on Response quality before it’s complete.

Write instructions

Instructions is a free-text field, markdown supported, that shows up in the annotation workspace itself. Optional, but worth using for queue-wide guidance that applies across every item and label in the queue, rather than notes on a single label. For Support quality review, something like:

Rate the agent's final response, not the whole conversation.

- **Good**: fully answers the question and matches our tone guidelines
- **Needs work**: correct but incomplete, robotic, or missing context
- **Wrong**: factually incorrect or answers a different question than the one asked

Open Advanced settings

Advanced settings is collapsed by default and holds three fields:

• Assignment strategy: Manual is the only one you can pick today. Round Robin and Load Balanced are visible with a Coming soon chip, disabled
• Reservation timeout: how long an item stays reserved for the annotator who opened it, one of 15 minutes, 30 minutes, 1 hour (the default), or 4 hours. When it expires, the item is released back to the queue for another annotator to pick up
• Require reviewer approval: a gated feature that needs the review workflow entitlement. Turn it on and a fully annotated item lands in review instead of finishing outright, which Review submissions covers; without the entitlement, turning it on fails with an upgrade prompt

Save, then activate it

Click Create annotation queue. It saves in Draft, and Draft can only move to Active, nowhere else. Annotating doesn’t start until you make that move: open the queue’s row menu and click Activate.

Activating doesn’t add any items either: the queue starts empty, add them next. From Active, a queue can move to Paused and on to Completed; see Queue settings & limits for the exact transitions.

Dive deeper