Apollo Server 4 tracing with Future AGI

Trace GraphQL operations and resolvers in a self-hosted Future AGI Observe project using Apollo Server 4 and OpenTelemetry. This guide applies to the server that runs your resolvers, not Apollo Gateway or Router; do not install the GraphQL instrumentation on a gateway.

For model calls inside a resolver, add the traceAI instrumentor for that model library. GraphQL spans appear as unknown in Observe.

Set up

The pinned OpenTelemetry SDK requires Node.js matching ^18.19.0 || >=20.6.0.

npm install @apollo/server@4.13.0 graphql@16.14.2 \
  @opentelemetry/api@1.9.0 @opentelemetry/sdk-node@0.223.0 @opentelemetry/exporter-trace-otlp-http@0.223.0 \
  @opentelemetry/resources@2.12.0 @opentelemetry/instrumentation-graphql@0.71.0 @opentelemetry/instrumentation-http@0.223.0

Use these versions together with the companion example; its SDK setup relies on the tracing export from the pinned @opentelemetry/sdk-node version.

Set your self-hosted Future AGI origin and credentials:

export FI_BASE_URL=http://localhost:14338
export FI_API_KEY=...
export FI_SECRET_KEY=...
export FI_PROJECT_NAME=my-graphql-api

In your project, add instrumentation.mjs:

import { NodeSDK, tracing } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { SpanStatusCode } from "@opentelemetry/api";
import { GraphQLInstrumentation } from "@opentelemetry/instrumentation-graphql";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { FiSanitizingExporter } from "./fi-exporter.mjs"; // companion example

const baseUrl = (process.env.FI_BASE_URL ?? "https://api.futureagi.com").replace(/\/+$/, "");
const projectName = process.env.FI_PROJECT_NAME ?? "apollo-server-4";

const processor = new tracing.BatchSpanProcessor(
  new FiSanitizingExporter(
    new OTLPTraceExporter({
      url: `${baseUrl}/tracer/v1/traces`, // HTTP, not gRPC
      headers: { "X-Api-Key": process.env.FI_API_KEY ?? "", "X-Secret-Key": process.env.FI_SECRET_KEY ?? "" },
      compression: "none",
    }),
    { captureContent: process.env.FI_APOLLO_CAPTURE_CONTENT === "true" },
  ),
);

export const sdk = new NodeSDK({
  resourceDetectors: [], // no hostname, OS user or command line on the resource
  resource: resourceFromAttributes({
    project_name: projectName, // a RESOURCE attribute
    project_type: "observe",
    "service.name": projectName,
  }),
  spanProcessors: [processor],
  metricReaders: [],        // traces only
  logRecordProcessors: [],  // traces only
  instrumentations: [
    new HttpInstrumentation(),
    new GraphQLInstrumentation({
      depth: 1,
      allowValues: false,
      responseHook: (span, result) => {
        if (result?.errors?.length) span.setStatus({ code: SpanStatusCode.ERROR });
      },
    }),
  ],
});
sdk.start();

export const flushTelemetry = () => processor.forceFlush().catch(reportExportError);
export const shutdownTelemetry = () => sdk.shutdown().catch(reportExportError);
function reportExportError(error) {
  console.error(`telemetry export failed: ${error?.name ?? "Error"}${error?.code ? ` (${error.code})` : ""}`);
}

Add server.mjs. Import instrumentation before the application.

import { shutdownTelemetry } from "./instrumentation.mjs"; // 1. tracing first
import { createServer } from "./app.mjs";                  // 2. app.mjs imports @apollo/server
import { startStandaloneServer } from "@apollo/server/standalone";

const server = createServer(); // new ApolloServer({ typeDefs, resolvers, stopOnTerminationSignals: false })
const { url } = await startStandaloneServer(server, { listen: { port: 4000 } });

for (const signal of ["SIGINT", "SIGTERM"]) {
  process.once(signal, async () => {
    await server.stop();
    await shutdownTelemetry(); // flushes the last batch
    process.exit(0);
  });
}

Get fi-exporter.mjs and the sample app.mjs from the companion example and place them beside these files. The exporter filters query and error text; see Privacy.

Run node server.mjs, then send a request to http://localhost:4000/, for example:

curl -X POST http://localhost:4000/ -H 'content-type: application/json' \
  -d '{"query":"query Profile { profile(id: \"1\") { name } }"}'

Open Observe, select your project, and open the resulting trace.

What each setting is for

Setting	Why
Instrumentation imported before @apollo/server	The instrumentations patch graphql and http when those load. Apollo loaded first leaves graphql unpatched: the server answers, and Observe gets no GraphQL spans at all.
X-Api-Key and X-Secret-Key	Both are required. Apollo sets neither, the exporter does.
project_name on the resource	A span attribute with that name does not count. Without the resource attribute the batch is dropped. project_type=observe too.
compression: "none"	Optional. Future AGI cloud and self-hosted v1.56.0 and later also accept gzip; earlier self-hosted releases answered it with HTTP 400. This exporter sends none by default, the line keeps it that way.
resourceDetectors: []	The NodeSDK default adds host.name, process.owner (the OS user name) and process.command_args (local file paths) to every batch.
metricReaders: [], logRecordProcessors: []	NodeSDK starts OTLP metric and log exporters unless told not to. The Future AGI endpoint takes traces only, and the defaults stall shutdown by several seconds.
stopOnTerminationSignals: false	Apollo’s built-in SIGINT/SIGTERM handler stops the server and then re-raises the signal, which ends the process before your handler can flush. With the default, the last batch can be lost.
depth: 1	Bounds the field spans, see below.
allowValues: false	The instrumentation masks literal values in the query text. Variable values are never recorded by this version either way.
responseHook	Marks the operation span ERROR when the result carries GraphQL errors, see Errors.

Field spans

depth: 1 limits resolver spans to top-level fields. Nested fields do not have their own spans at this depth, so errors appear on the operation span. Increase the depth for nested resolver timings; this can create a span per field per list item. ignoreTrivialResolveSpans does not reduce the span count in this Server 4 setup.

Resolver context

Inside a resolver, trace.getActiveSpan() returns the graphql.resolve span, and a span you start there is its child in the same trace. You do not pass a span through the GraphQL context.

import { trace } from "@opentelemetry/api";
const tracer = trace.getTracer("my-app");
const profiles = new Map([["1", { name: "Ada" }]]);

const resolvers = {
  Query: {
    profile: (_parent, { id }) =>
      tracer.startActiveSpan("load-profile", async (span) => {
        try { return profiles.get(id) ?? null; } finally { span.end(); }
      }),
  },
};

With depth: 1 this holds for top-level resolvers. For a nested resolver below the depth cap there is no resolve span, so the active span there is the nearest one above it.

What you see in Observe

• One trace per HTTP request: POST (method and status code), graphql.parse and graphql.validate (absent when Apollo’s cache already has the operation), the operation span named query <OperationName> (query when the client sent no name), and the field span.
• Operation name, type, field name, path and type as attributes. Operation span names come from the instrumentation, not from a graphql.execute span as older Apollo pages show.
• One extra trace at startup holding a single graphql.parseSchema span.
• Your own spans from the resolver, exactly as you made them.

Errors

The instrumentation marks the resolve span ERROR when a resolver throws, but it leaves the operation span UNSET. A GraphQL server also answers HTTP 200 with an errors array, so the POST span is not ERROR either. The responseHook above sets the operation span to ERROR whenever the result has errors, including partial results (data plus errors). The ERROR status carries no message.

Privacy

Default: structure only. In the export for this recipe, with the defaults:

Not exported	Exported
Variable values (never recorded by the instrumentation)	Operation name and type
Inline literals in the query text	Field name, path, return type, parent type
The query text (graphql.source)	http.request.method, http.response.status_code
The error message, the stack trace, the span status message	exception.type on a failing resolve span
URL, host name, user agent, peer address	Span names, timing, status code
Host name, OS user, command line, process id on the resource	project_name, project_type, service.name

fi-exporter.mjs does this by allowlist on the spans of the two instrumentations. Spans you create yourself are not touched: what you put on them is your decision.

Opt in with FI_APOLLO_CAPTURE_CONTENT=true. That keeps the query text (literals stay masked by allowValues: false) and the error message. The stack trace and variable values stay out. An error message can contain whatever your resolver put in it, including paths and user data, so turn this on only for debugging.

Flush and shutdown

The batch processor exports every 5 seconds. On SIGINT/SIGTERM, stop the server and call sdk.shutdown() to flush the last batch. If the Future AGI endpoint is unreachable or answers 401, flush and shutdown reject. The example catches that and logs the error class so telemetry never takes the server down.

One server, one key pair, one project. Do not pick the project from a request header. Sampling is the customer’s decision: the default sampler keeps every trace, and a ratio below 1 drops traces on purpose.

Apollo Server 5

This recipe targets Apollo Server 4. With Server 5, the HTTP POST span can be missing, leaving GraphQL spans in separate traces. Do not rely on this recipe for complete Server 5 HTTP tracing.

Troubleshooting

Symptom	Cause and fix
No traces in the project	project_name is missing from the resource, or only one of the two keys is set. A span attribute is not a substitute. Check the exporter URL ends in /tracer/v1/traces and, on a self-hosted release before v1.56.0, that the body is not gzip.
Traces have a POST span but no GraphQL spans	@apollo/server was imported before ./instrumentation.mjs. Put the tracing import first.
Last requests missing after a restart	Apollo’s default signal handler ended the process. Set stopOnTerminationSignals: false and shut down the SDK yourself.
Shutdown takes several seconds	A default OTLP metric or log exporter is running. Set metricReaders: [] and logRecordProcessors: [].
Too many spans	depth is unset. ignoreTrivialResolveSpans does not help on Server 4.
The operation span is not red when a query fails	Add the responseHook. The stock instrumentation only marks the field span.
Query text or an error message shows up in Observe	FI_APOLLO_CAPTURE_CONTENT is true, or you do not use fi-exporter.mjs.
Two traces for one request	The POST span is missing: Apollo Server 5 (see above), or http was loaded before ./instrumentation.mjs.