Falcon AI Trace Debugging

Falcon AI does its work by reading your agent’s traces: a trace is the structured record of one request, broken into spans for each LLM call, tool invocation, or sub-step inside it. The agent has to be sending traces to Future AGI before any of the next steps can run.

Three lines below set that up. OpenAIInstrumentor patches the OpenAI SDK so every API call is captured automatically. The @tracer.agent decorator on your agent’s entry point makes each request appear as one parent span with the OpenAI calls nested underneath.

from fi_instrumentation import register, FITracer
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="research-assistant-demo",
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)
tracer = FITracer(trace_provider.get_tracer("research-assistant-demo"))

from openai import OpenAI

client = OpenAI()


# Replace this with your own agent's entry point.
# The @tracer.agent decorator makes each call show up as one parent span
# in your Future AGI Tracing project, with the OpenAI calls nested underneath.
@tracer.agent(name="my_agent")
def my_agent(user_message: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a research assistant. Provide citations to support your claims."},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content


# Asking for citations on a topic the model has no search tool for is a
# common failure mode (the model fabricates papers from training data).
# This gives Falcon AI a failing trace to analyze in the next step.
print(my_agent("What's the seminal paper on transformers?"))
print(my_agent("What are the key papers on contrastive learning for self-supervised vision?"))

trace_provider.force_flush()

You should see two responses printed, and a new trace on your project’s Tracing page in the dashboard. For broader instrumentation patterns (custom spans, metadata tagging, prompt template tracking), see Manual Tracing.

Falcon AI picks up whatever page you’re viewing as context. Open it on a trace detail page and the trace ID auto-attaches as a context chip in the chat input, so every question and skill in this conversation answers against that specific trace.

In Tracing, click into the failing trace so the trace detail page is the active view. Open the Falcon AI sidebar and type:

What went wrong with this trace?

You should see a plain-English diagnosis: the model fell back to parametric memory and invented paper descriptions instead of grounding its answer in real sources.

The context chip on the chat input is what scopes every answer to this trace

Same conversation. The skill /analyze-trace-errors classifies issues against an error taxonomy (Hallucinated Content, Tool Misuse, Wrong Intent, etc.), assigns a severity to each finding, and produces a quality scorecard for the trace.

/analyze-trace-errors

You should see Hallucinated Content returned as a High impact finding (the model invented papers from training data instead of grounding the answer in retrieved sources), plus a quality scorecard and recommended fixes.

The severity assigned to each finding is what turns a wall of trace text into a triage list

This is diagnosis with suggestions. The next turn converts that suggestion into a paste-ready diff.

The third turn invokes /fix-with-falcon, which reads the system prompt and model output from the trace’s LLM span and returns a copy-pasteable prompt edit in a Current / Replace with format. The Current block is pulled directly from the span so the diff is grounded in what the agent actually saw, not guessed from a description.

/fix-with-falcon

You should see a diff that keeps the original system prompt and appends a refusal instruction, so the agent declines to answer rather than invent citations when it has no grounded source.

The Current block is pulled straight from the trace’s LLM span, not retyped from memory

Paste the Replace with block into my_agent’s system message, then re-run the same query that originally exposed the failure:

print(my_agent("What's the seminal paper on transformers?"))
trace_provider.force_flush()

Open the new trace on the Tracing page and run /analyze-trace-errors on it from Falcon AI again.

You should see a clean refusal in the response instead of a confidently invented citation list, and the second /analyze-trace-errors run should return no Hallucinated Content finding for this trace.