Annotate Datasets with Human-in-the-Loop Workflows

Create the annotation view from inside the tracing project whose spans you want to annotate, not from a standalone dataset. A view created off a project-scoped trace carries that project with it; a view created outside a project doesn’t, and get_labels(project_id=...) (which log_annotations() calls under the hood) only returns labels attached to a project, so labels created without one are invisible to the SDK.

1. Go to app.futureagi.com → Tracer (left sidebar under OBSERVE) and open the project with the spans you want to annotate
2. Open any trace, then open its annotation drawer
3. From the drawer, click Create New View and give it a descriptive name, e.g. “Response Quality Review”

You should see the view editor open with Static Fields, Response Fields, and Labels sections ready to configure.

Static fields give annotators read-only context. Response fields hold the output they’ll judge.

• Under Static Fields, select the columns that provide reference context (e.g. user_query, context). Annotators see these but can’t edit them
• Under Response Fields, select the column with the model output to judge (e.g. response)

You should see both columns previewed in the view layout before you add labels.

Click Create New Label for each judgment you want annotators to make. Each label needs a name, an annotation type, and (for non-categorical types) a min/max range and a step size.

Field	Description
Name	The label’s name, shown to annotators and used to reference it from the SDK
Annotation Type	Categorical (predefined options), Numeric (a score on a scale), or Text (free-form feedback)
Description	Required guidance shown to annotators on how to apply the label
Display Options	Numeric labels only: currently offers a single option, Slider
Min / Max Value	Required for Numeric and Text labels: the lower and upper bounds of the score (for Text, these map to minimum/maximum character length)
Step Size	Numeric labels only, required: the increment the slider moves by

For this guide, create three labels:

Label name	Annotation Type	Description	Min	Max	Step Size
Sentiment	Categorical	Overall tone of the response	N/A	N/A	N/A
Relevance Score	Numeric	How well the response addresses the query	1	5	0.5
Reviewer Notes	Text	Free-form feedback or corrections	0	500	N/A

For the Sentiment label, define categories: “Positive”, “Negative”, “Neutral”.

Click Save to store the label.

You should see the three labels listed under Labels in the view editor.

1. In the view’s Annotators section, add the workspace members who should contribute annotations
2. Click Create to create the view
3. Each annotator opens the view, sees the static fields as read-only context and the response field alongside the label inputs, and applies labels row by row

You should see each labeled row’s status update as annotators work through the dataset. Changes save automatically.

Before pushing annotations programmatically, confirm the exact label names and the project you’re targeting. Annotation.log_annotations() matches label names by exact string, so a typo or case mismatch raises a ValueError and aborts the call.

import os
from fi.annotations import Annotation

client = Annotation(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

projects = client.list_projects()
for p in projects:
    print(f"  {p.name} (id: {p.id}, type: {p.project_type})")

labels = client.get_labels(project_id=projects[0].id)
for label in labels:
    print(f"  {label.name}: type {label.type} (id: {label.id})")

Expected output:

  My Tracing Project (id: proj_abc123, type: observe)
  Sentiment: type categorical (id: lbl_001)
  Relevance Score: type numeric (id: lbl_002)
  Reviewer Notes: type text (id: lbl_003)

For bulk annotation or CI pipelines, push a pandas DataFrame with Annotation.log_annotations(). Each row references a traced span by its context.span_id, and each annotation column follows annotation.{label_name}.{type}. The {label_name} segment must match a label name from the previous step exactly, including case and spaces.

import os
import pandas as pd
from fi.annotations import Annotation

client = Annotation(
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

# Column format: annotation.{label_name}.{type}
# Types: label (categorical), score (numeric), text, rating (1-5 stars), thumbs (True/False)
df = pd.DataFrame({
    "context.span_id": ["span_abc123", "span_def456", "span_ghi789"],
    "annotation.Sentiment.label": ["Positive", "Negative", "Neutral"],
    "annotation.Relevance Score.score": [4.5, 2.0, 3.5],
    "annotation.Reviewer Notes.text": [
        "Accurate and well-structured response",
        "Hallucinated a date that wasn't in the context",
        "Correct but could be more concise",
    ],
    "annotation.notes": [
        "Reviewed by QA team",
        "Flagged for retraining",
        None,
    ],
})

result = client.log_annotations(df, project_name="My Tracing Project")

print(f"Annotations created: {result.annotationsCreated}")
print(f"Annotations updated: {result.annotationsUpdated}")
print(f"Notes created:       {result.notesCreated}")
print(f"Errors:              {result.errorsCount}")

Expected output:

Annotations created: 9
Annotations updated: 0
Notes created:       2
Errors:              0

Try it with a mismatched label name to see the guardrail in action. Rename annotation.Sentiment.label to annotation.sentiment.label (lowercase) and rerun:

df_bad = df.rename(columns={"annotation.Sentiment.label": "annotation.sentiment.label"})
client.log_annotations(df_bad, project_name="My Tracing Project")

ValueError: No annotation label found for name 'sentiment' and type 'label' in project 'My Tracing Project'

Fix it by copying the exact name from the get_labels() output in Step 5 (Sentiment, not sentiment) and rerun with the original DataFrame.