Falcon AI End-to-End Workflow

Falcon AI reads your agent’s traces: the structured record of one request, broken into spans for each LLM call, tool invocation, or sub-step inside it. OpenAIInstrumentor patches the OpenAI SDK so every call is captured automatically, and @tracer.agent wraps your agent’s entry point so each request lands as one parent span with the OpenAI calls nested underneath.

from fi_instrumentation import register, FITracer
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="falcon-ai-end-to-end",
)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)
tracer = FITracer(trace_provider.get_tracer("falcon-ai-end-to-end"))

from openai import OpenAI

client = OpenAI()


# Replace this with your own agent's entry point.
# @tracer.agent makes each call show up as one parent span in your
# Future AGI Tracing project, with the OpenAI calls nested underneath.
@tracer.agent(name="my_agent")
def my_agent(user_message: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a customer support assistant for an electronics store. Answer questions about products and orders."},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content


# A support agent with no grounding tool tends to fabricate specifics
# (tracking numbers, return windows, warranty lengths) when asked about
# them. That gives Falcon AI a failing trace to analyze next.
print(my_agent("Where is order ORD-12345?"))
print(my_agent("What's your return policy for opened wireless headphones?"))

trace_provider.force_flush()

You should see two new traces in Tracing → your project within a few seconds. If nothing appears, see Troubleshooting below.

Falcon AI picks up whatever page you’re viewing as context, so opening the sidebar from your project’s Tracing page scopes every question and skill to that project automatically.

/analyze-trace-errors runs across every trace in the project, classifies each issue against an error taxonomy (Hallucinated Content, Wrong Intent, Tool Misuse, and others), and scores every trace 1 to 5.

Stay on the Tracing page, open the sidebar, and type:

Analyze trace errors in this project

You should see a completion card listing per-trace scores and the dominant error category. Switch to the Feed tab in Tracing to see the same findings per-trace, with the quote that triggered each one.

Same conversation. /build-dataset reads the findings from the previous turn and writes the matching rows to a new dataset. This locks the bad traces as a regression dataset, a fixed snapshot you can re-run anytime: when you try a fix later, you score it against the exact same failing inputs instead of new traffic that may not reproduce the same problem.

Build me a dataset called falcon-demo-failures with the queries from the traces flagged with Hallucinated Content. Columns: query (text), agent_output (text), context (text), failure_category (text).

You should see a completion card with a link to the new dataset. Open Datasets → falcon-demo-failures to confirm the rows.

Same conversation. /run-evaluations runs Future AGI evals (LLM-as-judge metrics like factual_accuracy or completeness) against every row in the dataset and returns per-row and aggregate scores. This is the baseline the fix needs to beat.

Run factual_accuracy and completeness evals on the falcon-demo-failures dataset.

You should see factual_accuracy low and completeness high: the agent fully addresses each question, but the answers are invented.

/fix-with-falcon reads the system prompt and model output from a specific span and returns a copy-pasteable prompt edit in a Current / Replace with format. Unlike the previous skills it needs a single failing trace as context, not a whole project, so open it from a trace detail page.

For ungrounded hallucinations like these, the typical fix is a refusal instruction: the agent is told to decline rather than invent specifics when it lacks tool grounding.

Open one of the worst-scoring traces from the Feed. With that trace as context, type:

/fix-with-falcon

You should see sections for What happened, Root cause in the agent, The fix (current vs replace with), and Expected score improvement.

Paste the Replace with block as your new system prompt and re-run the same queries through your traced agent. Back in Falcon AI:

Re-run the same evals on falcon-demo-failures and compare to the previous run.

Sample after-fix scores, illustrative, your numbers will vary:

You should see factual_accuracy recover because the agent no longer fabricates, while completeness stays high because the refusal still addresses the question.