Traces Are Noisy or Incomplete
Traces you don't recognize show up in this project, or a trace's Type/Cost/Session/User is blank even though the LLM calls inside it ran. Usual causes: a process-wide instrumentor, a hand-built root span, or an oversized payload.
Symptom
A trace shows up, but something about it is wrong in one of two directions: either it’s not yours, or it’s missing the fields that make it useful. Both usually trace back to how the root span (the top row of the trace) was created, not to a bug in your data.
- Rows appear in this project that don’t match your own code’s input/output shape: a different call signature, a different kind of prompt.
- A trace’s Type shows “unknown”, Cost shows $0.00, and it’s missing from both the Sessions and Users tabs, even though you can see LLM calls with a real model and cost when you open the span tree underneath it.
- A trace is missing entirely, or your exporter logs a
413 Request Entity Too LargeorRESOURCE_EXHAUSTED.
Quick checks
- Does more than one workflow using the same framework (e.g. LangChain) run inside the same process? Instrumenting one wires up all of them.
- Did you create the top-level span yourself (
tracer.start_as_current_span(...)) instead of letting the framework’s auto-instrumentor create the root? - Does any single call send an unusually large prompt, tool result, or document as an attribute?
Causes and fixes
| Cause | What you see | Fix |
|---|---|---|
| One instrumentor patches the whole process | Traces from code you don’t own show up in this project | SomeFrameworkInstrumentor().instrument(tracer_provider=...) patches that framework’s callback/client machinery for the entire process, not just the module that called it. A second instrument() call elsewhere in the same process is a no-op and reuses the first tracer_provider. Give unrelated workflows their own process, or their own tracer_provider registered before either one runs. |
| Suppressing drops spans, it doesn’t redirect them | You suppressed the unrelated calls and now they don’t show up anywhere, not just in this project | suppress_tracing() drops spans created in its block. It never moves them to another project. If that code still needs to be observable, give it its own tracer_provider/project instead of suppressing it. |
| Manual root span never gets the standard fields | Trace Type is “unknown”, no Session/User | An auto-instrumented span picks up session.id/user.id from using_session/using_user automatically. A root span you create yourself doesn’t get gen_ai.span.kind, session.id, or user.id for free: set them explicitly on it (see below). |
| The row’s Cost/Model always reflect the root span | Cost and Model stay $0.00 / blank on the row even after Session and User are fixed | The trace table’s Cost, Model, and Tokens columns read the root span only. A hand-built root that never sets those attributes shows blank on the row even though the LLM call underneath it has both; open the span tree to see them there. |
| Oversized payload | Exporter logs 413 Request Entity Too Large / RESOURCE_EXHAUSTED, or a span silently never lands | A single span over 16 MiB is rejected outright. Cap attribute size with OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT or span_limits so no one attribute can grow that large. |
Fix it: set the standard fields on a manual root span
from fi_instrumentation import using_session, using_user
from fi_instrumentation.fi_types import FiSpanKindValues, SpanAttributes
with using_session(session_id), using_user(user_id):
with tracer.start_as_current_span("agent_turn") as span:
result = agent.invoke({"messages": [{"role": "user", "content": question}]})
span.set_attribute(SpanAttributes.GEN_AI_SPAN_KIND, FiSpanKindValues.AGENT.value)
span.set_attribute(SpanAttributes.SESSION_ID, session_id)
span.set_attribute(SpanAttributes.USER_ID, user_id)
Set all three explicitly, even though using_session/using_user also tag a FITracer span automatically; a plain OpenTelemetry tracer does not get them, so setting them here works either way.
Fix it: keep an unrelated workflow out of this project
from fi_instrumentation import suppress_tracing
with suppress_tracing():
result = other_workflow.invoke(...) # never traced, anywhere, in any project
Only reach for this if that workflow doesn’t need to be observable at all. If it does, register a second tracer_provider for it instead: see Instrument your project.
async with suppress_tracing(): isn’t supported: it raises a TypeError and can leave tracing suppressed past the block. Use the synchronous with, even inside an async def.
Fix it: cap oversized attributes
export OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=100000
or in code, once at registration:
from fi_instrumentation import register, SpanLimits
trace_provider = register(
project_name="my_project",
span_limits=SpanLimits(max_attribute_length=100000),
)
Diagnostic commands
Print each span’s name and attributes as they’re created, so you can see exactly which span has what before anything is exported:
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, SpanExporter
class PrintAttributes(SpanExporter):
def export(self, spans):
for span in spans:
print(span.name, dict(span.attributes))
return 0
trace_provider.add_span_processor(SimpleSpanProcessor(PrintAttributes()))
If the root span’s print shows no gen_ai.span.kind/session.id/user.id, it never had them; that’s the manual-root case, not a UI bug. If you see spans printed for code you don’t recognize, that’s the shared-instrumentor case.
Minimal smoke test
Run one request through end to end, then check three things on that trace: it’s the only new row in the project (no unrelated code’s calls alongside it), its Type/Session/User are populated on the row, and it landed at all if the request included an unusually large input or output.
Escalate
If a trace still looks wrong after the fixes above, contact support@futureagi.com with the project_name, the trace ID, and which of the three symptoms above you’re still seeing.
Prevent recurrence
- Give each distinct framework-based workflow its own process or its own
tracer_providerbefore you run more than one in the same service. - If you build a root span by hand, treat
gen_ai.span.kind,session.id, anduser.idas a fixed checklist to set on it every time, not a maybe. - Set a size cap once at registration, rather than after you hit the 16 MiB limit in production.
Next steps
Questions & Discussion