Building Golden Datasets from Production Traces with Falcon AI

Falcon AI reads your agent’s traces, so the agent has to be sending traces to Future AGI before any later step can run. OpenAIInstrumentor patches the OpenAI SDK so every API call is captured automatically. The @tracer.agent decorator on your agent’s entry point makes each classification appear as one parent span Falcon AI can filter on.

from fi_instrumentation import register, FITracer
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="email-triage-prod",
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)
tracer = FITracer(trace_provider.get_tracer("email-triage-prod"))

from openai import OpenAI

client = OpenAI()


# Replace this with your own agent's entry point.
# The @tracer.agent decorator makes each call show up as one parent span
# in your Future AGI Tracing project, with the OpenAI calls nested underneath.
@tracer.agent(name="triage_email")
def triage_email(email_text: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Classify this email into one of: urgent, billing, technical, general, spam. Reply with just the category name."},
            {"role": "user", "content": email_text},
        ],
    )
    return response.choices[0].message.content


# A classifier with a thin prompt will misclassify ambiguous emails (hostile tone
# over a small issue, multi-issue emails, etc.). Run a varied batch so Falcon AI
# has both clean classifications and likely misclassifications in the next step.
print(triage_email("Production is down. Payment processing has been failing for 30 minutes."))
print(triage_email("WORST SERVICE EVER. I have been on hold for 2 hours. CALL ME BACK."))
print(triage_email("I have a billing question and also my login is not working since yesterday."))
print(triage_email("Why am I being charged $499 when I signed up for the $49 plan? Please fix this or I am canceling."))

trace_provider.force_flush()

You should see four printed category labels and no errors. force_flush() blocks until the spans reach Future AGI, so once it returns, the traces are visible in your project’s Tracing tab. For broader instrumentation patterns see Manual Tracing.

Falcon AI picks up whatever page you’re viewing as context. Open it from your project’s Tracing page (the context chip should show the project name), then type:

What categories did my agent assign across these traces, and which ones look like misclassifications?

You should see a category histogram and a list of traces where the category looks off given the email content (your wording and counts will vary). These flagged misclassifications are a strong starting point, not ground truth. You’ll confirm them in a later step.

/build-dataset reads the traces in context and writes matching rows to a new dataset. The skill follows whatever selection criteria you give it, so the prompt below bakes in a coverage rule that mixes easy-pass rows with the misclassifications from the previous turn.

/build-dataset

Build a dataset called email-triage-eval-v1. Pull rows from the traces in this project. Selection criteria: include at least 2 traces from each category (urgent, billing, technical, general, spam) plus the likely misclassifications you flagged in the previous turn. Total target: 12-15 rows. Columns:

• email_text (text): the email body the agent classified
• predicted_category (text): what the agent chose
• trace_id (text): so we can trace any failure back

You should see a completion card linking to the new dataset, with 12-15 rows and every category represented. A dataset that is 90% successes won’t catch regressions; one that is 90% failures won’t catch false positives. The “at least 2 from each category plus the misclassifications” rule gives both classes meaningful coverage.

predicted_category is what the agent chose. To turn the dataset into an eval, you need expected_category, what the agent should have chosen. For genuinely ambiguous rows (hostile tone over a small issue, multi-issue emails) there is no single correct answer, so this step uses a NEEDS_REVIEW value plus a review_note column to surface them for human judgment instead of poisoning the eval with arbitrary labels.

Add a column expected_category (text) to email-triage-eval-v1. For each row, propose the correct category based on the email text. For rows where the correct category is genuinely ambiguous (e.g., hostile tone over a small issue, multi-issue emails), use the value NEEDS_REVIEW and add a one-sentence note in a new column review_note (text) explaining why.

You should see both columns populated on every row, with a split between confident expected_category values and a few rows tagged NEEDS_REVIEW. Open the dataset in Datasets → email-triage-eval-v1, click each NEEDS_REVIEW row, and decide based on your team’s routing rules. Edit the rows in the UI or ask Falcon AI to update them.

/run-evaluations runs an eval template from your workspace’s catalog against every row in the dataset and returns per-row and aggregate scores. Describe the goal in plain English so Falcon AI picks the right template (here, an exact-match check between two text columns).

/run-evaluations

Run an evaluation on email-triage-eval-v1 that checks whether predicted_category exactly matches expected_category for each row. Use the eval template from this workspace that best fits a string-equality check between two columns.

You should see a per-row pass/fail/skip verdict and an aggregate pass rate that is neither 0% nor 100%. Both the pass pattern and the fail pattern are what you want: a regression test where every row passes is not testing anything, and one where every row fails is just noisy. The dataset now has compounding value: any future prompt change can be re-scored against it in one chat message.