Instrument and Verify: Add Tracing to an Existing App and Prove It Worked

Install

Everything new lands in one directory. Existing files get a dependency, some configuration, one call at the entry point, and one around the model call.

your-repo/
├── observability/futureagi/
│   ├── setup.py               # provider, exporter, instrumentor      G1 G2
│   ├── fi_verify.py           # the checker, downloaded below         all ten
│   ├── futureagi_spans.py     # the attribute helpers                 G4 G5 G10
│   └── futureagi_rollup.py    # the model on a stream, your own rates G8
├── app/main.py                # + one scope at the entry point        G3 G6 G7
├── requirements.txt           # + fi-instrumentation-otel
└── .gitignore                 # + .fi_verify/

If your repository ships as a package, put observability/futureagi/ under your own package root instead of the top level.

Every step below offers two tracks, the Future AGI SDK and plain OpenTelemetry. Pick one and stay on it from here to the end. The two tracks write files of the same name that are not interchangeable, so every listing names its own track on the first line.

# Future AGI SDK track
pip install fi-instrumentation-otel      # not fi-instrumentation. Python 3.11+
pip install traceai-openai               # one per framework in use: -openai-agents, -anthropic,
                                         # -langchain, -llamaindex, -crewai, -litellm, -bedrock, ...

# OpenTelemetry track
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Then the checker. One file, no dependency beyond the OpenTelemetry SDK, and Step 1 runs it.

mkdir -p observability/futureagi
curl -fsSL https://docs.futureagi.com/fi_verify.py -o observability/futureagi/fi_verify.py
shasum -a 256 observability/futureagi/fi_verify.py
# 75ce8d9cb5bafa1cbeea42affe8b464433ab97e438a53410f27678855d484de7

export FI_API_KEY="your-api-key"
export FI_SECRET_KEY="your-secret-key"
export FI_PROJECT_NAME="my-app"

What a verified integration means

These failures all produce a trace list that looks populated, and reading the dashboard cannot tell them apart from a correct integration:

• Cost and tokens are missing everywhere, because the LLM spans carry neither a model nor a token count and nothing can price them
• One request appears as three traces, and each piece looks valid on its own
• Sessions and users are blank, so conversations do not group and per-customer spend cannot be read
• Evals report nothing, which looks identical to an eval that found no problems
• Nothing arrives at all, and the client stays healthy because the collector refused the batch quietly

So the result is decided by a checker rather than by looking. Ten gates, each tied to the step that closes it:

Nine gates read a local capture of your spans. G1 reads two receipts written during the run, because a capture says nothing about whether anything arrived.

Tutorial

python observability/futureagi/fi_verify.py preflight

# observability/futureagi/setup.py     Future AGI SDK track
import os, sys
from opentelemetry import trace
from fi_instrumentation import FITracer, register
from fi_instrumentation.fi_types import ProjectType
from traceai_openai import OpenAIInstrumentor

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))   # for fi_verify

# Import once at process start, AFTER whatever loads your .env and before any model call.
# Imported earlier the keys are not there yet, and only G1 says so.
tracer_provider = None
tracer = trace.get_tracer(__name__)   # a no-op tracer, so no key means no spans, not a crash

if os.getenv("FI_API_KEY") and os.getenv("FI_SECRET_KEY"):
    tracer_provider = register(
        project_name=os.getenv("FI_PROJECT_NAME", "my-app"),
        project_type=ProjectType.OBSERVE,
        set_global_tracer_provider=True,
    )
    OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
    # FITracer, not get_tracer(): a plain Tracer drops session.id and user.id, failing G6 and G7
    tracer = FITracer(tracer_provider.get_tracer(__name__))
    if os.getenv("FI_VERIFY") == "1":
        import fi_verify; fi_verify.attach(tracer_provider)

# observability/futureagi/setup.py     OpenTelemetry track
import os, sys, contextvars
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider, SpanProcessor
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))   # for fi_verify

# Import this AFTER whatever loads your .env, or the keys are not there yet and only G1 says so.
# On the RESOURCE: without it the collector answers 400 and the client still looks healthy
resource = Resource.create({"project_name": os.getenv("FI_PROJECT_NAME", "my-app"),
                            "project_type": "observe"})
provider = TracerProvider(resource=resource)

if os.getenv("FI_API_KEY") and os.getenv("FI_SECRET_KEY"):
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
        endpoint="https://api.futureagi.com/tracer/v1/traces",
        headers={"X-Api-Key": os.getenv("FI_API_KEY"),
                 "X-Secret-Key": os.getenv("FI_SECRET_KEY")})))

# Step 3 sets this once at the edge; every span picks it up here, so no call site remembers it
_scope = contextvars.ContextVar("fi_scope", default={})

class FiScope(SpanProcessor):     # the base class no-ops the other three methods
    def on_start(self, span, parent_context=None):
        for k, v in _scope.get().items(): span.set_attribute(k, v)

provider.add_span_processor(FiScope())
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

if os.getenv("FI_VERIFY") == "1":
    import fi_verify; fi_verify.attach(provider)

# your entry point: the scope outside, the root inside     Future AGI SDK track
from fi_instrumentation import using_attributes
from observability.futureagi.setup import tracer

# The scope goes OUTSIDE the root. Opened within it, it reaches the children and misses
# the root itself, and G6 and G7 read every span in the trace.
def handle(message):                # an HTTP route, a consumer, a job, a CLI run
    with using_attributes(session_id=conversation_id, user_id=account_id,
                          tags=["prod"], metadata={"tenant": tenant}):
        with tracer.start_as_current_span("orders.reprice") as root:
            root.set_attribute("gen_ai.span.kind", "CHAIN")
            root.set_attribute("input.value", message.body)
            answer = run_the_work(message)      # every model call nests under this
            root.set_attribute("output.value", answer)
            return answer

# your entry point: the scope outside, the root inside     OpenTelemetry track
from observability.futureagi.setup import tracer, _scope

# Set here and nowhere else. FiScope copies it onto every span in the request, root included.
def handle(message):                # an HTTP route, a consumer, a job, a CLI run
    token = _scope.set({"session.id": conversation_id, "user.id": account_id})
    try:
        with tracer.start_as_current_span("orders.reprice") as root:
            root.set_attribute("gen_ai.span.kind", "CHAIN")
            root.set_attribute("input.value", message.body)
            answer = run_the_work(message)      # every model call nests under this
            root.set_attribute("output.value", answer)
            return answer
    finally:
        _scope.reset(token)

export FI_MODEL_RATES='{"llama-3.3-70b-versatile": [0.59, 0.79]}'   # {"<model>": [in, out]} per 1M

# observability/futureagi/futureagi_spans.py     Future AGI SDK track
from contextlib import contextmanager
from observability.futureagi.futureagi_rollup import MODEL

@contextmanager               # wrap the client call. The instrumentor writes the rest
def llm_call(model):
    t = MODEL.set(model)
    try: yield
    finally: MODEL.reset(t)

# a local TOOL or RETRIEVER span, and the root: three lines each
with tracer.start_as_current_span("lookup_price") as span:
    span.set_attribute("gen_ai.span.kind", "TOOL")
    span.set_attribute("gen_ai.tool.name", "lookup_price")
    span.set_attribute("output.value", json.dumps(catalogue.price(sku)))

# observability/futureagi/futureagi_rollup.py     Future AGI SDK track
import json, os, contextvars
from opentelemetry import context as otel_context
from opentelemetry.sdk.trace import SpanProcessor
from observability.futureagi.setup import tracer_provider as P   # None until the keys are set

RATES = json.loads(os.getenv("FI_MODEL_RATES", "{}"))   # {"<model>": [in, out]} per 1M
MODEL = contextvars.ContextVar("fi_model", default=None)   # what llm_call is about to call
RUN = otel_context.create_key("fi_run")     # rides the OpenTelemetry context, so a copied
M = "gen_ai.request.model"                  # context carries it over a thread hand-off too.
T = ["gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens",
     "gen_ai.usage.total_tokens", "gen_ai.cost.total"]

class RollUp(SpanProcessor):
    def on_start(self, span, parent_context=None):   # the instrumentor never leaves its LLM
        if MODEL.get(): span.set_attribute(M, MODEL.get())   # span current in your code, so
    def on_end(self, span):                          # the name goes on as the span opens
        run, a = otel_context.get_value(RUN), span.attributes or {}
        if run is not None:                    # None outside a request: nothing to sum onto
            if T[0] in a:                      # tokens land on LLM spans, which end first
                i, o = RATES.get(a.get(M), [0, 0])          # your rate, for a model
                a = dict(a, **{T[3]: (a[T[0]] * i + a.get(T[1], 0) * o) / 1e6})   # we do not price
            for k in T: run[k] = run.get(k, 0) + a.get(k, 0)

# register() leaves _default_processor set, and the first add_span_processor call on that
# provider discards the exporter it installed. Clearing it first is what keeps delivery.
if P: P._default_processor = False; P.add_span_processor(RollUp())

# observability/futureagi/futureagi_spans.py     OpenTelemetry track
# Every key comes from here. One module, so a name can be misspelled only once.
import json
from contextlib import contextmanager

def _set(span, attrs):              # None is never written: an empty attribute
    for k, v in attrs.items():      # reads as a missing one
        if v is not None: span.set_attribute(k, v if isinstance(v, (str, bool, int, float))
                                             else json.dumps(v, default=str))

@contextmanager
def span_of(t, name, kind, opening, alias=None):    # one shape for all twelve kinds
    with t.start_as_current_span(name) as span:
        _set(span, {"gen_ai.span.kind": kind, **opening})
        yield lambda out=None, extra=None: _set(span, {"output.value": out, **(extra or {}),
                                                       **({alias: out} if alias else {})})

# observability/futureagi/futureagi_rollup.py     OpenTelemetry track
import json, os
from opentelemetry import context as otel_context
from opentelemetry.sdk.trace import SpanProcessor
from observability.futureagi.setup import provider as P   # the provider Step 2 built

RATES = json.loads(os.getenv("FI_MODEL_RATES", "{}"))   # {"<model>": [in, out]} per 1M
RUN = otel_context.create_key("fi_run")     # rides the OpenTelemetry context, so a copied
M = "gen_ai.request.model"                  # context carries it over a thread hand-off too.
T = ["gen_ai.usage.input_tokens", "gen_ai.usage.output_tokens",
     "gen_ai.usage.total_tokens", "gen_ai.cost.total"]

class RollUp(SpanProcessor):
    def on_end(self, span):        # the base class no-ops the other three methods
        run, a = otel_context.get_value(RUN), span.attributes or {}
        if run is not None and T[0] in a:      # tokens land on LLM spans, which end first
            i, o = RATES.get(a.get(M), [0, 0])            # your rate, for a model
            a = dict(a, **{T[3]: (a[T[0]] * i + a.get(T[1], 0) * o) / 1e6})   # we do not price
            for k in T: run[k] = run.get(k, 0) + a.get(k, 0)

P.add_span_processor(RollUp())

# your entry point, around the root from Step 3
run = {k: 0 for k in T}          # mutated in place, so a copied context shares it
token = otel_context.attach(otel_context.set_value(RUN, run))
try:
    with tracer.start_as_current_span("orders.reprice") as root:
        answer = plan_and_run(order_id)
        run[T[2]] = run.get(T[2]) or run.get(T[0], 0) + run.get(T[1], 0)   # only if the
        for k in T: root.set_attribute(k, run.get(k, 0))              # provider sent none
finally:
    otel_context.detach(token)

printf '\n.fi_verify/\n' >> .gitignore     # echo would join the last line
export FI_VERIFY=1

python observability/futureagi/fi_verify.py preflight   # keys, route, three broken controls
python -m your_app                                      # one real request, your entry point
python observability/futureagi/fi_verify.py check       # ten gates: receipts, then spans

  PASS  G1   preflight ok, delivery ok
  PASS  G2   project_name='my-app' project_type='observe'
  PASS  G3   7 spans, 1 trace(s), 1 root(s), 0 orphan(s)
  PASS  G4   3 LLM span(s), 0 untyped
  PASS  G5   prompt and completion on every LLM span
  PASS  G6   session.id=['c-4182']
  PASS  G7   user.id=['acct-993']
  PASS  G8   model on every LLM span
  PASS  G9   3 LLM span(s): 3 priceable from the model and tokens, 0 carrying your own cost
  PASS  G10  no credential in any span attribute

  Future AGI integrated
  GREEN LIGHT achieved

The same six steps on a real repository

Everything above was run against openai/openai-agents-python, on its examples/customer_service airline support agent. Nothing in that repository was written for this guide, which is the point: it is a normal app with normal problems.

It is a fair target because it has all four of them at once. It emits no trace the platform can read. A single message fans out across a triage agent, a handoff, a specialist agent and a local tool, so Step 2 has real work to do. main.py is an interactive REPL, so there is no entry point that runs once, which is Step 2’s third case. And it carries a conversation id and a passenger context already, so Step 3 has a real session and a real user to attach rather than invented ones.

git clone --depth 1 https://github.com/openai/openai-agents-python
cd openai-agents-python
pip install openai-agents fi-instrumentation-otel traceai-openai-agents

observability/futureagi/ is the four files from Step 2 and Step 4 on the Future AGI SDK track, with traceai-openai-agents as the instrumentor. The one file added outside it is the entry point the repository does not have. No business logic was edited, and main.py was imported, not modified.

# examples/customer_service/run_traced.py
"""One customer message through the airline support agent, traced end to end.

main.py is an interactive REPL, so the repository has no entry point that runs once. This
is that entry point: the roll-up and the scope outside, the CHAIN root inside, a flush
before returning. Run it twice with the same FI_SESSION_ID to see a session of two traces.
"""
import asyncio, os, sys, uuid

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))

from observability.futureagi.setup import tracer, tracer_provider
from observability.futureagi import futureagi_rollup as R
from fi_instrumentation import using_attributes
from opentelemetry import context as otel_context

from agents import Runner, RunConfig, set_default_openai_api
from examples.customer_service.main import AirlineAgentContext, triage_agent

if os.getenv("OPENAI_BASE_URL"):      # an OpenAI-compatible endpoint that is not OpenAI
    set_default_openai_api("chat_completions")   # only OpenAI serves the Responses API

MODEL = os.getenv("AGENT_MODEL", "gpt-4o-mini")


async def handle(question, session_id, user_id):
    run = {k: 0 for k in R.T}          # mutated in place, so a copied context shares it
    token = otel_context.attach(otel_context.set_value(R.RUN, run))
    try:
        with using_attributes(session_id=session_id, user_id=user_id,
                              tags=["prod"], metadata={"channel": "web"}):
            with tracer.start_as_current_span("support.turn") as root:
                root.set_attribute("gen_ai.span.kind", "CHAIN")
                root.set_attribute("input.value", question[:8000])
                root.set_attribute(R.M, MODEL)   # the Model column reads the root, not the
                                                 # LLM spans the instrumentor writes it on
                result = await Runner.run(
                    triage_agent, [{"content": question, "role": "user"}],
                    context=AirlineAgentContext(), run_config=RunConfig(model=MODEL))
                answer = str(result.final_output)
                root.set_attribute("output.value", answer[:8000])
                run[R.T[2]] = run.get(R.T[2]) or run.get(R.T[0], 0) + run.get(R.T[1], 0)
                for k in R.T: root.set_attribute(k, run.get(k, 0))
                return result, answer
    finally:
        otel_context.detach(token)


async def main():
    question = " ".join(sys.argv[1:])
    session_id = os.getenv("FI_SESSION_ID") or "conv_" + uuid.uuid4().hex[:12]
    user_id = os.getenv("ACCOUNT_ID", "acct_10427")

    result, answer = await handle(question, session_id, user_id)
    print(f"{result.last_agent.name}: {answer}")

    if tracer_provider: tracer_provider.force_flush()   # a batch sends on a timer


if __name__ == "__main__":
    asyncio.run(main())

export FI_VERIFY=1 FI_SESSION_ID=conv_94ac3e74fcc4

python observability/futureagi/fi_verify.py preflight
python examples/customer_service/run_traced.py "How much baggage am I allowed to bring on the plane?"
python observability/futureagi/fi_verify.py check

  PASS  G1   preflight ok, delivery ok
  PASS  G2   project_name='support-agent-quickstart' project_type='observe'
  PASS  G3   13 spans, 1 trace(s), 1 root(s), 0 orphan(s)
  PASS  G4   3 LLM span(s), 0 untyped
  PASS  G5   prompt and completion on every LLM span
  PASS  G6   session.id=['conv_94ac3e74fcc4']
  PASS  G7   user.id=['acct_10427']
  PASS  G8   model on every LLM span
  PASS  G9   3 LLM span(s): 3 priceable from the model and tokens, 0 carrying your own cost
  PASS  G10  no credential in any span attribute

  Future AGI integrated
  GREEN LIGHT achieved

Run it a second time with the same FI_SESSION_ID and a second message, and the two turns join one session.

What that produced

The trace list reads the root and nothing else, which is why Step 4 puts the totals there. Latency and status arrive on their own; the tokens, the cost and the model are on the root because this run put them there.

Two support.turn traces, the Cost column reading the total this run rolled onto each root because our table does not price this model

Inside a trace, the instrumentor typed the agents and the model calls, and the handoff. faq_lookup_tool is a local function the instrumentor cannot see, so its TOOL span is the three lines from Step 4. The attributes panel is the same list Step 4 sends, read back off a real span.

One support.turn trace: the CHAIN root, the triage agent, the handoff, the FAQ agent, the local faq_lookup_tool span and the three LLM spans, with the attributes panel open on a real span

Because session.id was set once at the edge, every turn in the conversation groups under one session without any turn knowing about the other.

One session, grouped by the session.id Step 3 set once at the edge

The evals bound to it

Each of these binds only to an attribute the run above already carries, read out of .fi_verify/spans.jsonl.

On an instrumented LLM span, input.value holds only the first message and the full turn sits under gen_ai.input.messages.*. Bind anything that reads the request at Traces scope against the root, where input.value is the customer’s actual question.

If your app is not Python

The listings translate line for line, and everything reaches the same endpoint over plain OpenTelemetry. Carry across:

• Both headers, X-Api-Key and X-Secret-Key, on https://api.futureagi.com/tracer/v1/traces
• project_name and project_type on the resource, not the span
• One root per unit of work
• The scope from Step 3, set at the edge
• The Step 4 attribute keys, byte for byte
• The roll-up, only if you need the trace list’s Cost column or your own rates

Spring Boot also needs management.tracing.sampling.probability=1.0. First-party SDKs and the full framework list are in the instrumentor reference.

The checker still runs. preflight and check are plain Python with no dependency at all, so they work next to an app in any language. Only fi_verify.attach() is Python-bound, because it installs a processor inside your process. Without it, write the capture yourself: one JSON object per line in .fi_verify/spans.jsonl, each with name, trace_id, span_id, parent_id (null on the root), attrs, and resource. That is roughly ten lines in any OpenTelemetry span exporter, and check reads it the same way either way.

Troubleshooting

Keep that one trace across a service boundary with Distributed Tracing.