Introducing Annotation Queues
We're introducing Annotation Queues - a new way to scale human judgement and domain expertise via in-app human review.
These past few months, we've been deep in conversation with teams scaling AI agents across specific domains: financial services, healthcare, insurance, etc. One pattern keeps coming up: the gap between automated evaluation and human judgment.
Automated evals catch a lot. But there's a category of decisions—compliance edge cases, nuanced quality issues, domain-specific judgment calls—where you need a human in the loop. The challenge isn't whether to do human review. It's how to do it systematically without burning out your domain experts or missing the examples that actually matter.
Most teams start with ad-hoc processes: exporting CSVs, sharing screenshots in Slack, manually hunting through logs for examples to review. This works early on, but breaks down as agent complexity and volume increase.
Today, we're launching Annotation Queues: a way to organize and manage logs that require human review, labeling, or quality assessment. Queues provide an automated workflow for routing specific events to your team for annotation, replacing scattered processes with a centralized system.
What's New
Manual Queue Creation
When you need precise control over what gets reviewed, you can manually add events to a queue directly from the Log Store. This approach is useful when curating a specific set of events for review or dealing with edge cases that need immediate attention.
Here's how it works:
- Navigate to the Log Store in your project
- Apply filters to identify the events you want to add to the queue
- Select the events you want to include (you can select all matching events)
- Click the Add to dropdown menu
- Select Add to Queue
This gives you point in time curation for specific needs: investigating a production incident, preparing for a compliance audit, or building out a golden dataset for evaluation.
.png)
Automated Queue Creation
Manual curation works for targeted needs, but it doesn't scale for continuous annotation workflows. With automated queue creation, you can set up rules that automatically add matching events to a queue without manual intervention.
.png)
There are two ways to set this up:
Option 1: During Manual Selection. If you're already in the Log Store filtering events, you can turn that filter into an automation rule. Apply your filters, and when creating the queue, toggle the Queue automation checkbox. Your filters will be saved as automation rules going forward.
Option 2: From the Annotations Tab. Navigate to Annotations, click Create Queue, set up your filter criteria, toggle the Queue automation checkbox, and save.
From that point on, any new events matching your criteria flow into the queue automatically.
Use Cases
Annotation queues support a range of human review workflows:
- Quality Assurance: Route low confidence predictions or edge cases for human review. When your automated evaluators flag potential issues, those events can flow directly to QA reviewers, focusing expert attention on outputs that actually need scrutiny.
- Active Learning: Identify and label examples where your model is uncertain. When agents show low confidence or require multiple retry attempts, routing them to domain experts helps you systematically improve where it matters most.
- Compliance Review: Flag sensitive or regulated content for manual verification. In industries like financial services and healthcare, certain categories of agent output require human sign off. Automated queues ensure coverage on high risk interactions without relying on manual filtering.
- Training Data Curation: Collect and label examples to improve your datasets. Annotation queues give domain experts a structured way to review and label production examples, replacing spreadsheet workflows.
- Performance Monitoring: Sample production traffic for ongoing quality assessment. Automated queues can route a percentage of traffic to reviewers on an ongoing basis, maintaining visibility into production quality.
Getting Started
We recommend starting with one queue that addresses your most pressing need.
If you have a specific set of events that need immediate attention, start with manual queue creation. Filter your Log Store, select the relevant events, and add them to a queue.
If you need continuous coverage without manual intervention, set up automated rules from the Annotations tab. Define your filter criteria, toggle automation, and let matching events flow in automatically.
Once you have events in a queue, you can set up human evaluator fields with custom criteria to structure your annotation workflows. Learn more in our human evaluators documentation.
