Human Annotation
Human annotation lets people on your team review AI output and record scores. Use it to build a quality baseline, check that LLM-as-a-Judge scores agree with human judgement, and label cases for datasets.
There are two ways to annotate:
- Directly. Open a trace, observation or session and select Annotate.
- In a queue. Collect items in an annotation queue and work through them one at a time.
Human Annotation is not in the sidebar today. Open it at /project/<projectId>/annotation-queues, or use the link in the confirmation shown after you add items to a queue. Among the built-in roles, only Owner includes annotation queue and score access today. See Members & Roles.
Set up a queue
Create score configs
In Project Settings > Scores / Evaluation, add a score config for each thing reviewers should rate: numeric (with optional minimum and maximum), categorical (with labels) or boolean. See Score configs.
Create the queue
On the Human Annotation page, select New queue and fill in:
- Name
- Description (optional): instructions for reviewers.
- Score Configs: the scores reviewers fill in for every item.
- Advanced Settings: assign users to the queue.
The number of queues per project depends on your plan.
Add items
Add traces, observations or sessions to the queue:
- In the Tracing table, select traces and choose Add to Annotation Queue from the bulk actions. The traces are added in the background.
- On a trace, observation or session, use the queue menu next to Annotate to add that item to a queue. Items already in a queue show In queue(s).
Annotate
Open the queue and work through the items. For each item the reviewer sees the trace, observation or session, fills in the queue's scores, can leave a comment, and completes the item to move on. The queue shows how many items are pending.
Queues are filled by hand. There is no rule that adds items to a queue automatically.
Annotate without a queue
Open any trace, observation or session and select Annotate. Pick score configs and enter values. The scores appear on the item next to API and evaluator scores.
Using annotation scores
- Compare human scores with LLM-as-a-Judge scores on the same traces to check your evaluators.
- Filter traces by an annotation score to find bad answers, then add them to a dataset.
- Chart annotation scores on a custom dashboard.
API
Annotation queues, queue items and assignments are also available through the public API under /api/public/annotation-queues. See API Reference.
Tips
- Write clear guidance in the score config description and the queue description, for example what a 1 and a 5 mean.
- Start with a small queue to check that reviewers read the criteria the same way.
- Have two reviewers score the same items now and then to check agreement.