Scores

What a score is

A score is one answer, from a person or a system, to one label about one source. It’s the single record type behind every judgement in Annotation, whether it came from working a queue item directly or from an inline score submitted against a source.

A score carries the annotator’s answer to the label, who or what produced it, and where it came from, and it looks the same regardless of which surface wrote it.

What makes a score unique

What decides whether a new judgement becomes its own score or lands on an existing one is a key made of four fields. Change any one of them and you get a different score, not an update to an old one.

• Source: the trace or item the score is about
• Label: the label being answered
• Annotator: the person account that submitted it, whether they worked a queue item or scored it inline
• Queue item: the queue item the score was submitted through, if any

This key applies to scores that have an annotator; a score written without one keys differently: see Scores with no annotator below. The field names above are conceptual: for their exact names in the API and SDK, see Data models.

Say priya@yourteam.com scores a trace’s Response quality label. Two cases show what the key decides.

Two queues, two scores. Priya works the trace’s item in the Support quality review queue and scores it. That’s Score A. The same trace later lands in a second queue, Escalation audit, and she scores it again with the same label. That’s Score B. Source, label, and annotator match between A and B, but the queue item doesn’t: Support quality review on A, Escalation audit on B. Two submissions, two independent scores, neither overwrites the other. The same person’s judgement through two different queues is two separate pieces of evidence, not a correction.

Two writes, one queue item. Now say Priya instead scores that same trace on Response quality inline, then does it again the same way. Both writes resolve to the same queue item, the source’s default one, so all four fields match this time: source, label, annotator, and queue item are identical between the two writes. That’s Score C: the second write lands on it and updates it, instead of creating a new one.

flowchart TD
accTitle: How a score's key decides whether two writes merge or stay independent
accDescr: Two writes to the same trace and Response quality label, submitted through different queue items, become two independent scores, A and B. Two writes sharing the same default queue item, with matching source, label, and annotator, merge instead: the second write lands on the same score, C, and its old value is kept in C's history.
SRC["Source · one trace"]
LBL["Label · Response quality"]
ANN["Annotator · priya@yourteam.com"]
QI1["Queue item · Support quality review"]
QI2["Queue item · Escalation audit"]
QI3["Queue item · source's default queue"]
SRC --> SC1["Score A"]
SRC --> SC2["Score B"]
SRC --> SC3["Score C"]
LBL --> SC1
LBL --> SC2
LBL --> SC3
ANN --> SC1
ANN --> SC2
ANN --> SC3
QI1 --> SC1
QI2 --> SC2
QI3 --> SC3
W2["Second inline write · same key"] -->|merges, old value kept in history| SC3

Scores with no annotator

A score written without an annotator uses a narrower key: source and label alone, with no queue item and no annotator in it. That key guarantees at most one such score can exist for a given source and label: see What a score remembers below for what happens when its value changes.

What a score remembers

Editing a score’s value doesn’t erase what was there before. The prior value is appended to the score’s history before the new one is written, so the current answer and the trail of what it used to be both live on the same record, visible in the queue’s annotation panel.

Every score also carries a score_source tag recording where it came from: human or api. It’s a record of origin you can read back, not a setting you choose.

A score is its own record

A score doesn’t depend on the queue item it came from. The queue item it names is provenance, a record of which pass through which queue produced it, not something the score needs to keep existing. That’s why an inline score and a queue score show up side by side in the same list: they’re the same kind of record, and the queue item is optional context on either one.

Why it matters

• Scoring the same source in more than one queue never collides, so a support audit and a compliance audit can run over the same traces independently
• Correcting a score is safe: the old value doesn’t disappear, it’s superseded on the same record
• Every view of a source, and every export, shows one list of scores no matter which surface wrote them

Keep exploring