Distributed Tracing: Propagate Traces Across Services with OpenTelemetry

Propagate OpenTelemetry trace context across two services with W3C TraceContext headers, so a gateway call and its downstream LLM call land in one trace.

📝
TL;DR

Run a gateway service and a backend service that calls Gemini, propagate the W3C traceparent header between them, and see both services’ spans land in one trace on your Future AGI dashboard instead of two disconnected ones.

TimeDifficultyPackage
20 minIntermediatefi-instrumentation-otel
Prerequisites
  • Future AGI account: app.futureagi.com
  • API keys: FI_API_KEY and FI_SECRET_KEY (see Get your API keys)
  • Google Gemini API key (GOOGLE_API_KEY) for the backend’s LLM call
  • Python 3.11 (Node 20+, JDK 17+, or .NET 8+ if you follow the TypeScript, Java, or C# tab)

Install

pip install fi-instrumentation-otel traceAI-google-genai flask requests python-dotenv google-genai
export FI_API_KEY="your-fi-api-key"
export FI_SECRET_KEY="your-fi-secret-key"
export GOOGLE_API_KEY="your-google-api-key"
npm install @traceai/fi-core @traceai/google-genai @google/generative-ai @opentelemetry/api @opentelemetry/core @opentelemetry/instrumentation express
export FI_API_KEY="your-fi-api-key"
export FI_SECRET_KEY="your-fi-secret-key"
export GOOGLE_API_KEY="your-google-api-key"
<dependency>
    <groupId>ai.traceai</groupId>
    <artifactId>traceai-java-core</artifactId>
    <version>1.0.0</version>
</dependency>
export FI_API_KEY="your-fi-api-key"
export FI_SECRET_KEY="your-fi-secret-key"
export GOOGLE_API_KEY="your-google-api-key"
dotnet add package OpenTelemetry OpenTelemetry.Exporter.OpenTelemetryProtocol OpenTelemetry.Instrumentation.AspNetCore OpenTelemetry.Instrumentation.Http DotNetEnv
export FI_API_KEY="your-fi-api-key"
export FI_SECRET_KEY="your-fi-secret-key"
export GOOGLE_API_KEY="your-google-api-key"

Tutorial

Set the same propagator on both services

Both the gateway and the backend need to agree on how a trace ID gets encoded into an HTTP header. Set the W3C TraceContext propagator globally on each process before you create any spans.

from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator

set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),  # W3C traceparent header
    W3CBaggagePropagator(),           # W3C baggage header
]))
import { propagation } from "@opentelemetry/api";
import { W3CTraceContextPropagator } from "@opentelemetry/core";

propagation.setGlobalPropagator(new W3CTraceContextPropagator());

The Java SDK registers the W3C TraceContext propagator by default, so there’s no separate call to make here: TraceAI.initFromEnvironment() in Step 2 sets it globally when it initializes the SDK.

Sdk.SetDefaultTextMapPropagator(new CompositeTextMapPropagator(
    new TextMapPropagator[] {
        new TraceContextPropagator(),
        new BaggagePropagator()
    }));

You should see no output yet, this only configures how context gets encoded. Run it on both services, not just one.

Register both services under the same project

Call register() on the gateway and the backend with an identical project_name. That shared name is how Future AGI groups spans from two separate processes into one trace view.

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType

provider = register(
    project_name="distributed_tracing_demo",
    project_type=ProjectType.OBSERVE,
    set_global_tracer_provider=True,
)
import { register, ProjectType } from "@traceai/fi-core";

const tracerProvider = register({
  projectName: "distributed_tracing_demo",
  projectType: ProjectType.OBSERVE,
  setGlobalTracerProvider: true,
});
import ai.traceai.TraceAI;
import ai.traceai.FITracer;

TraceAI.initFromEnvironment();
FITracer tracer = TraceAI.getTracer();
builder.Services.AddOpenTelemetry()
    .WithTracing(tracerBuilder =>
    {
        tracerBuilder
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService(serviceName: serviceName, serviceVersion: "1.0.0"))
            .AddSource(serviceName)
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri("https://api.futureagi.com/tracer/v1/traces");
                opts.Protocol = OtlpExportProtocol.HttpProtobuf;
                opts.Headers = $"X-Api-Key={fiApiKey},X-Secret-Key={fiSecretKey}";
            });
    });

Both processes now export to the same Future AGI project. Nothing lands until you create a span.

Auto-instrument the backend's Gemini call

On the backend only, instrument the Gemini client so every call becomes a span with the model name, token counts, and input/output attached automatically.

from traceai_google_genai import GoogleGenAIInstrumentor
from google import genai
import os

GoogleGenAIInstrumentor().instrument(tracer_provider=provider)
client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])
import { GoogleGenAIInstrumentation } from "@traceai/google-genai";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { GoogleGenerativeAI } from "@google/generative-ai";

registerInstrumentations({
  tracerProvider,
  instrumentations: [new GoogleGenAIInstrumentation()],
});

const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!);

No auto-instrumentor is wired up in this recipe. The Java backend calls Gemini over plain HTTP and wraps that call in a manual Backend.GeminiCall span in Step 5 instead.

No auto-instrumentor is wired up in this recipe. The C# backend calls Gemini over plain HTTP and wraps that call in a manual Backend.GeminiCall activity in Step 5 instead.

@traceai/google-genai patches the @google/generative-ai package’s GoogleGenerativeAI class, not @google/genai. Import from @google/generative-ai or the call runs without a span. You should see the google-genai instrumentor register at startup, alongside the other OpenTelemetry setup logged by register().

Inject the trace context on the gateway's outgoing call

Before wiring this up, see what happens without it: run the two services as shown in Step 6 but with the inject(headers) line below removed (and the backend’s extract() in Step 5 removed too), then curl http://localhost:5100/ask. On the Future AGI dashboard you’ll see two separate root traces (one for Gateway.ProcessRequest, one for Backend.GeminiCall) because no traceparent header ever reaches the backend.

Fix it by writing the current span’s context into the outgoing request headers before the gateway calls the backend. question comes from the incoming request to the gateway’s own /ask route.

from opentelemetry import trace
from opentelemetry.propagate import inject
from flask import request
import requests as http_requests

tracer = trace.get_tracer(__name__)

question = request.args.get("question", "What is distributed tracing?")
with tracer.start_as_current_span("Gateway.CallBackend", kind=trace.SpanKind.CLIENT):
    headers = {}
    inject(headers)  # writes: traceparent: 00-<traceId>-<spanId>-01
    response = http_requests.post(
        "http://localhost:5101/generate",
        json={"question": question},
        headers=headers,
    )
const headers: Record<string, string> = { "Content-Type": "application/json" };
propagation.inject(context.active(), headers, {
  set: (carrier, key, value) => { carrier[key] = String(value); },
});
const response = await fetch("http://localhost:5101/generate", {
  method: "POST",
  headers,
  body: JSON.stringify({ question }),
});
Map<String, String> headers = new HashMap<>();
GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
    .inject(Context.current(), headers, Map::put);
// add headers to the outgoing HttpRequest
// AddHttpClientInstrumentation() injects traceparent automatically.
// Just make the call:
var response = await client.SendAsync(backendRequest);

inject() only sees a context if it’s called inside the active span’s block. Call it before the span exits, not after. You should see a traceparent value of the form 00-<32 hex>-<16 hex>-01 in the headers dict right after the inject() call.

Extract the trace context on the backend's incoming call

The backend reads the traceparent header back into a context, then makes that context active so any span it creates is a child of the gateway’s span. question comes from the JSON body the gateway just posted.

from opentelemetry import context
from opentelemetry.propagate import extract
from flask import request

question = request.json["question"]
ctx = extract(request.headers)
token = context.attach(ctx)
try:
    response = client.models.generate_content(model="gemini-2.0-flash", contents=question)
finally:
    context.detach(token)  # Flask reuses threads, always detach
const extractedCtx = propagation.extract(context.active(), req.headers, {
  get: (carrier, key) => {
    const val = carrier[key.toLowerCase()];
    return Array.isArray(val) ? val[0] : val;
  },
});

await context.with(extractedCtx, async () => {
  const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash" });
  const result = await model.generateContent(question);
  res.json({ answer: result.response.text() || "No response" });
});
Context extractedCtx = GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
    .extract(Context.current(), incomingHeaders, Map::get);

try (Scope scope = extractedCtx.makeCurrent()) {
    Span span = tracer.startSpan("Backend.GeminiCall", FISpanKind.LLM);
    try (Scope spanScope = span.makeCurrent()) {
        tracer.setInputValue(span, question);
        String answer = callGemini(question);
        tracer.setOutputValue(span, answer);
    } finally {
        span.end();
    }
}
// AddAspNetCoreInstrumentation() extracts traceparent automatically.
// Activity.Current already carries the gateway's TraceId here.
using var activity = activitySource.StartActivity("Backend.GeminiCall", ActivityKind.Client);

You should see the backend’s span carry the same trace ID the gateway printed, not a new one.

Start both services

The pieces from Steps 1-5 assemble into two small services: a backend on port 5101 that exposes /generate, and a gateway on port 5100 that exposes /ask and calls the backend. Save each file below, then start the backend first so the gateway doesn’t fail on connection errors.

# backend.py
import os
from flask import Flask, request, jsonify
from opentelemetry import context
from opentelemetry.propagate import set_global_textmap, extract
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_google_genai import GoogleGenAIInstrumentor
from google import genai

set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    W3CBaggagePropagator(),
]))

provider = register(
    project_name="distributed_tracing_demo",
    project_type=ProjectType.OBSERVE,
    set_global_tracer_provider=True,
)

GoogleGenAIInstrumentor().instrument(tracer_provider=provider)
client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])

app = Flask(__name__)

@app.route("/generate", methods=["POST"])
def generate():
    question = request.json["question"]
    ctx = extract(request.headers)
    token = context.attach(ctx)
    try:
        response = client.models.generate_content(model="gemini-2.0-flash", contents=question)
    finally:
        context.detach(token)  # Flask reuses threads, always detach
    return jsonify({"answer": response.text})

if __name__ == "__main__":
    app.run(port=5101)
# gateway.py
from flask import Flask, request, jsonify
from opentelemetry import trace
from opentelemetry.propagate import set_global_textmap, inject
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
import requests as http_requests

set_global_textmap(CompositePropagator([
    TraceContextTextMapPropagator(),
    W3CBaggagePropagator(),
]))

provider = register(
    project_name="distributed_tracing_demo",
    project_type=ProjectType.OBSERVE,
    set_global_tracer_provider=True,
)
tracer = trace.get_tracer(__name__)

app = Flask(__name__)

@app.route("/ask")
def ask():
    question = request.args.get("question", "What is distributed tracing?")
    with tracer.start_as_current_span("Gateway.ProcessRequest") as root_span:
        trace_id = format(root_span.get_span_context().trace_id, "032x")
        with tracer.start_as_current_span("Gateway.CallBackend", kind=trace.SpanKind.CLIENT):
            headers = {}
            inject(headers)  # writes: traceparent: 00-<traceId>-<spanId>-01
            response = http_requests.post(
                "http://localhost:5101/generate",
                json={"question": question},
                headers=headers,
            )
    return jsonify({"answer": response.json()["answer"], "traceId": trace_id})

if __name__ == "__main__":
    app.run(port=5100)
# Terminal 1
python backend.py
# Terminal 2
python gateway.py
// backend.ts
import express from "express";
import { context, propagation } from "@opentelemetry/api";
import { W3CTraceContextPropagator } from "@opentelemetry/core";
import { register, ProjectType } from "@traceai/fi-core";
import { GoogleGenAIInstrumentation } from "@traceai/google-genai";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { GoogleGenerativeAI } from "@google/generative-ai";

propagation.setGlobalPropagator(new W3CTraceContextPropagator());

const tracerProvider = register({
  projectName: "distributed_tracing_demo",
  projectType: ProjectType.OBSERVE,
  setGlobalTracerProvider: true,
});

registerInstrumentations({
  tracerProvider,
  instrumentations: [new GoogleGenAIInstrumentation()],
});

const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY!);

const app = express();
app.use(express.json());

app.post("/generate", async (req, res) => {
  const { question } = req.body;
  const extractedCtx = propagation.extract(context.active(), req.headers, {
    get: (carrier, key) => {
      const val = carrier[key.toLowerCase()];
      return Array.isArray(val) ? val[0] : val;
    },
  });

  await context.with(extractedCtx, async () => {
    const model = genAI.getGenerativeModel({ model: "gemini-2.0-flash" });
    const result = await model.generateContent(question);
    res.json({ answer: result.response.text() || "No response" });
  });
});

app.listen(5101, () => console.log("backend ready at :5101"));
// gateway.ts
import express from "express";
import { context, propagation, trace, SpanKind } from "@opentelemetry/api";
import { W3CTraceContextPropagator } from "@opentelemetry/core";
import { register, ProjectType } from "@traceai/fi-core";

propagation.setGlobalPropagator(new W3CTraceContextPropagator());

const tracerProvider = register({
  projectName: "distributed_tracing_demo",
  projectType: ProjectType.OBSERVE,
  setGlobalTracerProvider: true,
});
const tracer = trace.getTracer("gateway");

const app = express();

app.get("/ask", async (req, res) => {
  const question = (req.query.question as string) || "What is distributed tracing?";

  await tracer.startActiveSpan("Gateway.ProcessRequest", async (rootSpan) => {
    const traceId = rootSpan.spanContext().traceId;

    const answer = await tracer.startActiveSpan(
      "Gateway.CallBackend",
      { kind: SpanKind.CLIENT },
      async (callSpan) => {
        const headers: Record<string, string> = { "Content-Type": "application/json" };
        propagation.inject(context.active(), headers, {
          set: (carrier, key, value) => { carrier[key] = String(value); },
        });
        const response = await fetch("http://localhost:5101/generate", {
          method: "POST",
          headers,
          body: JSON.stringify({ question }),
        });
        const body = await response.json();
        callSpan.end();
        return body.answer;
      },
    );

    rootSpan.end();
    res.json({ answer, traceId });
  });
});

app.listen(5100, () => console.log("gateway ready at :5100"));
# Terminal 1
npx tsx backend.ts
# Terminal 2
npx tsx gateway.ts
// Backend.java
import ai.traceai.TraceAI;
import ai.traceai.FITracer;
import ai.traceai.FISpanKind;
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;

public class Backend {
    public static void main(String[] args) throws Exception {
        TraceAI.initFromEnvironment();
        FITracer tracer = TraceAI.getTracer();

        HttpServer server = HttpServer.create(new InetSocketAddress(5101), 0);
        server.createContext("/generate", exchange -> {
            Map<String, String> incomingHeaders = new HashMap<>();
            exchange.getRequestHeaders().forEach((k, v) -> incomingHeaders.put(k, v.get(0)));
            String question = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);

            Context extractedCtx = GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
                .extract(Context.current(), incomingHeaders, Map::get);

            String answer;
            try (Scope scope = extractedCtx.makeCurrent()) {
                Span span = tracer.startSpan("Backend.GeminiCall", FISpanKind.LLM);
                try (Scope spanScope = span.makeCurrent()) {
                    tracer.setInputValue(span, question);
                    answer = callGemini(question);
                    tracer.setOutputValue(span, answer);
                } finally {
                    span.end();
                }
            }

            byte[] body = answer.getBytes(StandardCharsets.UTF_8);
            exchange.sendResponseHeaders(200, body.length);
            exchange.getResponseBody().write(body);
            exchange.getResponseBody().close();
        });
        server.start();
        System.out.println("backend ready at :5101");
    }

    private static String callGemini(String question) throws Exception {
        String key = System.getenv("GOOGLE_API_KEY");
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=" + key))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(
                "{\"contents\":[{\"parts\":[{\"text\":\"" + question.replace("\"", "'") + "\"}]}]}"))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());
        return response.body();
    }
}
// Gateway.java
import ai.traceai.TraceAI;
import ai.traceai.FITracer;
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;

public class Gateway {
    public static void main(String[] args) throws Exception {
        TraceAI.initFromEnvironment();
        FITracer tracer = TraceAI.getTracer();

        HttpServer server = HttpServer.create(new InetSocketAddress(5100), 0);
        server.createContext("/ask", exchange -> {
            String question = "What is distributed tracing?";
            Span rootSpan = tracer.startSpan("Gateway.ProcessRequest");
            String traceId;
            String answer;
            try (Scope rootScope = rootSpan.makeCurrent()) {
                traceId = rootSpan.getSpanContext().getTraceId();
                Span callSpan = tracer.startSpan("Gateway.CallBackend", SpanKind.CLIENT);
                try (Scope callScope = callSpan.makeCurrent()) {
                    Map<String, String> headers = new HashMap<>();
                    GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
                        .inject(Context.current(), headers, Map::put);

                    HttpRequest.Builder builder = HttpRequest.newBuilder()
                        .uri(URI.create("http://localhost:5101/generate"))
                        .POST(HttpRequest.BodyPublishers.ofString(question, StandardCharsets.UTF_8));
                    headers.forEach(builder::header);

                    HttpResponse<String> response = HttpClient.newHttpClient()
                        .send(builder.build(), HttpResponse.BodyHandlers.ofString());
                    answer = response.body();
                } finally {
                    callSpan.end();
                }
            } finally {
                rootSpan.end();
            }
            byte[] body = ("{\"answer\": \"" + answer.replace("\"", "'") + "\", \"traceId\": \"" + traceId + "\"}")
                .getBytes(StandardCharsets.UTF_8);
            exchange.sendResponseHeaders(200, body.length);
            exchange.getResponseBody().write(body);
            exchange.getResponseBody().close();
        });
        server.start();
        System.out.println("gateway ready at :5100");
    }
}
# Terminal 1
java -jar backend.jar
# Terminal 2
java -jar gateway.jar
// Backend.cs
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry().WithTracing(t => t
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("backend"))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(opts =>
    {
        opts.Endpoint = new Uri("https://api.futureagi.com/tracer/v1/traces");
        opts.Protocol = OtlpExportProtocol.HttpProtobuf;
        opts.Headers = $"X-Api-Key={Environment.GetEnvironmentVariable("FI_API_KEY")}," +
                       $"X-Secret-Key={Environment.GetEnvironmentVariable("FI_SECRET_KEY")}";
    }));

var app = builder.Build();
var activitySource = new ActivitySource("backend");
var httpClient = new HttpClient();

app.MapPost("/generate", async (HttpContext ctx) =>
{
    using var reader = new StreamReader(ctx.Request.Body);
    var question = await reader.ReadToEndAsync();

    // AddAspNetCoreInstrumentation() extracts traceparent automatically;
    // Activity.Current already carries the gateway's TraceId here.
    using var activity = activitySource.StartActivity("Backend.GeminiCall", ActivityKind.Client);

    var key = Environment.GetEnvironmentVariable("GOOGLE_API_KEY");
    var payload = $"{{\"contents\":[{{\"parts\":[{{\"text\":\"{question.Replace("\"", "'")}\"}}]}}]}}";
    var response = await httpClient.PostAsync(
        $"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={key}",
        new StringContent(payload, Encoding.UTF8, "application/json"));
    return await response.Content.ReadAsStringAsync();
});

app.Run("http://localhost:5101");
// Gateway.cs
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry().WithTracing(t => t
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("gateway"))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSource("gateway")
    .AddOtlpExporter(opts =>
    {
        opts.Endpoint = new Uri("https://api.futureagi.com/tracer/v1/traces");
        opts.Protocol = OtlpExportProtocol.HttpProtobuf;
        opts.Headers = $"X-Api-Key={Environment.GetEnvironmentVariable("FI_API_KEY")}," +
                       $"X-Secret-Key={Environment.GetEnvironmentVariable("FI_SECRET_KEY")}";
    }));

var app = builder.Build();
var activitySource = new ActivitySource("gateway");
var httpClient = new HttpClient();

app.MapGet("/ask", async (HttpContext ctx) =>
{
    var question = "What is distributed tracing?";
    using var rootActivity = activitySource.StartActivity("Gateway.ProcessRequest");
    var traceId = rootActivity?.TraceId.ToString() ?? "";

    using var backendRequest = new HttpRequestMessage(HttpMethod.Post, "http://localhost:5101/generate")
    {
        Content = new StringContent(question, Encoding.UTF8, "text/plain"),
    };
    // AddHttpClientInstrumentation() injects traceparent automatically. Just make the call:
    var response = await httpClient.SendAsync(backendRequest);
    var answer = await response.Content.ReadAsStringAsync();

    return new { answer, traceId };
});

app.Run("http://localhost:5100");
# Terminal 1
dotnet run -- backend
# Terminal 2
dotnet run -- gateway

You should see each process print its own “ready at” line on ports 5100 and 5101.

Call the gateway and check the trace

curl http://localhost:5100/ask

You should see a JSON response with a traceId. Open the Future AGI dashboard, find the distributed_tracing_demo project, and open that trace ID. Gateway.ProcessRequest, Gateway.CallBackend, and the backend’s Gemini span all appear nested under one trace instead of as two separate traces.

Troubleshooting

SymptomCauseFix
Two separate traces instead of oneGateway and backend registered under different project_name valuesUse the exact same project_name string in both register() calls
traceparent header missing on the outgoing requestinject() called after the span’s with/try block exitedCall inject() while the span is still the active span
Backend spans show up as new root traces, not childrenExtracted context never made activeWrap the backend’s work in context.attach(ctx) (Python) or context.with(extractedCtx, ...) (TypeScript) before creating spans
TypeScript backend has no LLM span at allBackend imports @google/genai and calls GoogleGenAI, which @traceai/google-genai doesn’t patchImport GoogleGenerativeAI from @google/generative-ai instead
Java build fails to resolve traceai-java-corepom.xml uses groupId com.github.future-agi.traceAIUse groupId ai.traceai, version 1.0.0
Gateway throws a connection error on startupGateway started before the backend was listeningAlways start the backend first, then the gateway
Later requests on the same thread inherit the wrong parent span (Python)context.detach(token) skipped or not in a finally blockFlask reuses worker threads: always detach in finally

Continue with Manual Tracing to add custom spans inside either service without relying on auto-instrumentation.

Was this page helpful?

Questions & Discussion