What it is
Automation rules let you define conditions that automatically trigger actions on queue items — such as auto-adding items that match certain criteria or pre-filling label values based on span attributes. Instead of manually curating queue contents, you set the rules once and let matching items flow in automatically.How to set up an automation rule
Configure the rule
Fill in the rule configuration:
- Name — A descriptive rule name so your team knows what it does at a glance.
- Source Type — Which type of items this rule applies to (e.g., traces, spans).
- Conditions — Define match criteria:
- Field — The attribute to evaluate (e.g., span attribute, metric name).
- Operator — The comparison operator (equals, greater than, less than, contains).
- Value — The threshold or match string.
- Enabled — Toggle the rule on or off.
Preview and evaluate
Before enabling a rule in production, use these tools to validate it:- Preview — Click the Preview button to see which existing queue items would match the conditions without actually triggering any actions.
- Evaluate — The Evaluate action tests the rule against current items and shows detailed match results, so you can fine-tune conditions before going live.
Example rules
| Rule Name | Condition | Action |
|---|---|---|
| Flag low scores | eval_score < 0.5 | Auto-add to review queue |
| Long responses | token_count > 1000 | Auto-add for quality check |
| Error traces | status = “error” | Auto-add for analysis |
Automation rules are evaluated when new items are added to the queue. Existing items can be tested using the Evaluate action but are not retroactively processed unless you trigger evaluation manually.
Next steps
Create & Manage Queues
Set up the queues that your automation rules feed into.
Add Items to Queues
Learn about manual and programmatic ways to add items alongside automation.
Analytics & Agreement
Monitor the items your rules are adding and track annotation progress.