Get Inter-Annotator Agreement

Retrieve inter-annotator agreement metrics for a queue.

Get Inter-Annotator Agreement

Calculate inter-annotator agreement scores including Fleiss’ kappa and Cohen’s kappa.

GET https://api.futureagi.com/model-hub/annotation-queues/{id}/agreement/
Bearer 
          Click "Try It" to send a request

Authentication

HeaderDescription
X-Api-KeyYour API key
X-Secret-KeyYour secret key

Path Parameters

ParameterTypeRequiredDescription
idUUIDYesThe annotation queue ID

Response

FieldTypeDescription
fleiss_kappafloatFleiss’ kappa score for multi-annotator agreement (-1 to 1)
cohens_kappafloatCohen’s kappa score for pairwise agreement (-1 to 1)
agreement_by_labelobjectPer-label agreement scores keyed by label ID

Note

Agreement metrics require at least two annotators to have completed annotations on overlapping items. If insufficient data exists, values may be null.

Code Examples

import requests

queue_id = "your-queue-uuid"
resp = requests.get(
    f"https://api.futureagi.com/model-hub/annotation-queues/{queue_id}/agreement/",
    headers={"X-Api-Key": "YOUR_KEY", "X-Secret-Key": "YOUR_SECRET"}
)
data = resp.json()
print(f"Fleiss kappa: {data['fleiss_kappa']}")
import axios from 'axios';

const queueId = 'your-queue-uuid';
const resp = await axios.get(
  `https://api.futureagi.com/model-hub/annotation-queues/${queueId}/agreement/`,
  { headers: { 'X-Api-Key': 'YOUR_KEY', 'X-Secret-Key': 'YOUR_SECRET' } }
);
console.log(`Fleiss kappa: ${resp.data.fleissKappa}`);
curl "https://api.futureagi.com/model-hub/annotation-queues/your-queue-uuid/agreement/" \
  -H "X-Api-Key: YOUR_KEY" \
  -H "X-Secret-Key: YOUR_SECRET"
Was this page helpful?

Questions & Discussion