Annotate Datasets with Human-in-the-Loop Workflows

Navigate to the dataset you want to annotate and switch to the annotation interface.

1. Go to app.futureagi.com → Dataset (left sidebar under BUILD)
2. Click the name of the dataset you want to annotate
3. Click the Annotations tab at the top of the data table

An annotation view groups labels together and maps them to the columns annotators will see.

1. Click Create New View
2. Give the view a descriptive name (e.g., “Response Quality Review”)

Static Fields

Select the column(s) that provide read-only context to annotators (e.g., user_query, context). Annotators see these as reference material alongside the response but cannot edit them.

Response Fields

Select the column(s) containing the model output you want annotators to evaluate (e.g., response). This is the primary field annotators will judge and label.

Labels

3. Click New Label for each annotation type you need. Configure the following fields for each label:

Field	Description
Name	A clear label name (e.g., “Sentiment”, “Relevance Score”, “Reviewer Notes”)
Annotation Type	The input type: Categorical (predefined categories), Numeric (score on a scale), or Text (free-form feedback)
Description	Optional description to guide annotators on what the label means and how to apply it
Display Option	How the label renders in the annotation interface (e.g., dropdown or radio buttons for categorical; slider or number input for numeric)
Min / Max Value	For Numeric labels only — the lower and upper bounds of the score range (e.g., min: 1, max: 5)

For this guide, create three labels:

Label name	Annotation Type	Description	Min	Max
Sentiment	Categorical	Overall tone of the response	—	—
Relevance Score	Numeric	How well the response addresses the query	1	5
Reviewer Notes	Text	Free-form feedback or corrections	—	—

For the Sentiment categorical label, define categories: “Positive”, “Negative”, “Neutral”.

Annotators

4. In the Annotators section of the view, add workspace members who should contribute annotations. Each assigned annotator can open the view and apply labels to dataset rows.

5. Click Save to finalize the view.

1. In the Annotation View settings, find the Annotators section
2. Add workspace members who should contribute annotations
3. Each annotator opens the view, navigates through rows, and applies labels

Annotators see the static fields as read-only context and the response fields alongside the label inputs. Changes save automatically.

For bulk annotation or CI pipelines, use Annotation.log_annotations() to push annotations via a pandas DataFrame. Each row references a traced span by its context.span_id.

import os
import pandas as pd
from fi.annotations import Annotation

client = Annotation(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

# Build a DataFrame with annotation columns
# Column format: annotation.{label_name}.{type}
# Types: label (categorical), score (numeric), text, rating (1-5 stars), thumbs (True/False)
df = pd.DataFrame({
    "context.span_id": ["span_abc123", "span_def456", "span_ghi789"],
    "annotation.sentiment.label": ["Positive", "Negative", "Neutral"],
    "annotation.relevance.score": [4.5, 2.0, 3.5],
    "annotation.reviewer_notes.text": [
        "Accurate and well-structured response",
        "Hallucinated a date that wasn't in the context",
        "Correct but could be more concise",
    ],
    "annotation.notes": [
        "Reviewed by QA team",
        "Flagged for retraining",
        None,
    ],
})

result = client.log_annotations(df, project_name="My Tracing Project")

print(f"Annotations created: {result.annotationsCreated}")
print(f"Annotations updated: {result.annotationsUpdated}")
print(f"Notes created:       {result.notesCreated}")
print(f"Errors:              {result.errorsCount}")

Expected output:

Annotations created: 9
Annotations updated: 0
Notes created:       2
Errors:              0

Before logging annotations programmatically, verify which labels and projects exist.

import os
from fi.annotations import Annotation

client = Annotation(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

# List projects
projects = client.list_projects()
for p in projects:
    print(f"  {p.name} (id: {p.id}, type: {p.project_type})")

# List annotation labels for a specific project
labels = client.get_labels(project_id=projects[0].id)
for label in labels:
    print(f"  {label.name}: type {label.type} (id: {label.id})")

Expected output:

  My Tracing Project (id: proj_abc123, type: observe)
  sentiment: type categorical (id: lbl_001)
  relevance: type numeric (id: lbl_002)
  reviewer_notes: type text (id: lbl_003)