Request Logging and Observability in Agent Command Center

Request logs, Prometheus metrics, and OpenTelemetry trace export for the gateway

About

Agent Command Center logs every request and response, exports metrics to Prometheus, can export OpenTelemetry traces from self-hosted gateways, and propagates trace IDs for distributed tracing. Request logging is on by default, with no extra setup. OpenTelemetry export is off until you turn it on.


Request logging

Every request through Agent Command Center is logged with:

  • Request ID, trace ID, session ID
  • Model requested and model actually used
  • Provider that handled the request
  • Input/output token counts
  • Cost
  • Latency
  • Cache status (hit/miss/skip)
  • Guardrail results
  • Any errors or fallback events

Logs sync to the Future AGI dashboard automatically. View them in Agent Command Center > Logs.

Log levels

LevelWhat’s logged
errorFailed requests, provider errors, guardrail blocks
warnFallbacks, slow requests, budget warnings
infoEvery request (default)
debugFull request/response bodies, header details

For self-hosted deployments, set the log level in config.yaml:

logging:
  level: info

Distributed tracing

Agent Command Center propagates trace IDs across the request lifecycle. Set x-agentcc-trace-id on incoming requests and the same ID appears in all downstream provider calls and logs.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    trace_id="trace-from-my-app-abc123",
    user_id="user-42",
)

print(response.agentcc.trace_id)    # trace-from-my-app-abc123
print(response.agentcc.provider)    # openai
print(response.agentcc.latency_ms)  # 342
print(response.agentcc.cost)        # 0.00015
raw = client.chat.completions.with_raw_response.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={
        "x-agentcc-trace-id": "trace-from-my-app-abc123",
        "x-agentcc-user-id": "user-42",
    },
)
print(raw.headers.get("x-agentcc-trace-id"))
print(raw.headers.get("x-agentcc-cost"))
curl -i https://gateway.futureagi.com/v1/chat/completions \
  -H "Authorization: Bearer sk-agentcc-your-key" \
  -H "x-agentcc-trace-id: trace-from-my-app-abc123" \
  -H "x-agentcc-user-id: user-42" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
# Look for x-agentcc-trace-id in response headers

If you don’t set a trace ID, Agent Command Center generates one automatically. Use it for correlating gateway logs with your application logs.

OpenTelemetry export

Self-hosted gateways can also emit one OpenTelemetry span per request to any OTLP-compatible backend. Span attributes follow the gen_ai.* semantic conventions, and each span carries the gateway’s trace ID as agentcc.trace_id, so you can match it to its row in the request log above. The otel block in the gateway’s config.yaml turns export on; it’s off by default. Export never waits on the collector, so a slow or unreachable one doesn’t slow requests down.

Sending to your own Future AGI install? Go to Send gateway spans to self-hosted Observe.

otel:
  enabled: true
  exporter: otlp
  endpoint: "http://otel-collector:4318" # /v1/traces is appended when the URL has no path
  protocol: "http/protobuf"
  service_name: "agentcc-gateway"
  headers:                               # sent on every export request
    Authorization: "${OTEL_AUTH_HEADER}" # for example "Bearer <token>"

Warning

The gateway speaks OTLP over HTTP with protobuf only, so point it at your collector’s OTLP/HTTP port (usually 4318), not the gRPC port 4317. Setting enabled: true without exporter: otlp prints spans to stdout instead of exporting them.

With enabled: true and exporter: otlp, the gateway refuses to start when:

  • endpoint is missing, or isn’t an absolute http(s) URL
  • a header value is empty (usually a ${VAR} that isn’t set)
  • protocol is grpc or http/json

All otel keys:

KeyDefaultDescription
enabledfalseTurns span creation on
exporterstdoutstdout or otlp
endpointnoneCollector base URL; required for otlp
protocolhttp/protobufThe only supported value
headersnoneHeaders sent with every export, for collector authentication. Use ${VAR} to keep credentials out of the file
service_nameagentcc-gatewayservice.name resource attribute
sample_rate1.0Fraction of requests that get a span. 0 or unset is treated as 1.0; to stop spans, set enabled: false
attributesnoneExtra OTLP resource attributes
include_bodiesfalseAttach the prompt and completion (input.value / output.value), redacted with the organization’s privacy settings like the request log
trace_propagationfalseHonor an inbound W3C traceparent so the gateway span nests under the caller’s span
metadata_attributestrueAlso emit each caller metadata key as agentcc.metadata.<key>, next to the metadata JSON object
body_attributestrueEmit unknown scalar request-body fields as agentcc.body.<key> attributes
capture_headersnoneAllowlist of request headers to copy onto the span. Well-known credential headers (Authorization, Cookie, x-api-key and similar) are never copied

Send gateway spans to self-hosted Observe

When you self-host Future AGI alongside the gateway, point the otel block at your install’s fi-collector, the service that receives traces for Observe.

Add the block to the gateway’s config file:

  • Docker Compose: the file AGENTCC_CONFIG_PATH in .env names, relative to the repository root. If it’s still the bundled agentcc-gateway/config.example.yaml, copy it to agentcc-gateway/config.yaml and set AGENTCC_CONFIG_PATH=agentcc-gateway/config.yaml first (see LLM gateway)
  • Helm: under agentccGateway.config.otel, or in your own ConfigMap if you set agentccGateway.existingConfigMap

Create the API key and set its two variables (step 2) before you restart anything: the gateway won’t start while a ${VAR} in the block is unset.

otel:
  enabled: true
  exporter: otlp
  endpoint: "http://127.0.0.1:4318"   # see step 1 for your setup
  protocol: "http/protobuf"
  headers:
    X-Api-Key: "${AGENTCC_OTEL_API_KEY}"
    X-Secret-Key: "${AGENTCC_OTEL_SECRET_KEY}"
  attributes:
    project_name: "my-chatbot"        # Observe project the spans land in
  include_bodies: false               # see step 4
  trace_propagation: false            # see step 5
  1. Endpoint. fi-collector takes OTLP/HTTP on port 4318. Use the address the gateway can reach it on:

    Setupendpoint
    Standalone (docker-compose.yml)http://127.0.0.1:4318
    Distributed (docker-compose.distributed.yml)http://fi-collector:4318
    Helmhttp://<fullname>-fi-collector:4318

    In Standalone the gateway runs in the same app container as the collector. With Helm, <fullname> is futureagi for a release named futureagi, and kubectl get svc shows the exact Service name.

  2. API keys. The collector refuses an export without an API key pair. Create one in your own install (not app.futureagi.com) under Build > Keys, which Owners, Admins and workspace Admins can see. You get an API key and a secret key, and the secret key is shown only once, so copy both. Set them as AGENTCC_OTEL_API_KEY and AGENTCC_OTEL_SECRET_KEY:

    • Docker Compose: in .env
    • Helm: in a Secret listed under agentccGateway.extraEnvFrom

    Don’t reuse the names FI_API_KEY and FI_SECRET_KEY. Every service in the install reads .env, and the API and the simulation runner read those two names too, so they’d start using your key.

  3. Project. Spans land in the Observe project named by the project_name resource attribute, in the workspace the API key belongs to. If no project has that name yet, it’s created on the first export. Pick a name no other project in that workspace uses: projects are matched by name, so an existing project with that name gets the spans. Without a project_name, the collector refuses the export.

  4. Prompt and completion bodies. include_bodies is off by default, so spans carry metrics and metadata but no message content. Turning it on sends prompts and completions to Observe, redacted with your organization’s privacy settings, the same way the request log is. The request log has its own switch, logging.request_logging.include_bodies.

  5. Nesting under traceAI spans. If your application is instrumented with traceAI, sends its spans to the same project_name, and passes a W3C traceparent header on its calls to the gateway, set trace_propagation: true. The gateway span then nests under your application’s span instead of starting a second trace. Leave it off otherwise: any OpenTelemetry instrumentation in your application can send a traceparent, and the gateway span would then point at a parent Observe never receives, so the trace isn’t listed. Only a traceparent marked as sampled is honored.

  6. Apply and check. The gateway reads its config once, at startup, so restart it after every change:

    • Docker Compose, first time or after editing .env: docker compose up -d (docker compose restart doesn’t re-read .env)
    • Docker Compose, after editing only the config file: docker compose restart app in Standalone, or docker compose restart agentcc-gateway in Distributed (docker compose up -d doesn’t pick up a config-file change)
    • Helm: helm upgrade. If you use existingConfigMap, or changed only the Secret, also run kubectl rollout restart deployment/<fullname>-agentcc-gateway

    Send one request through the gateway, then open the my-chatbot project in Observe, in the workspace the API key belongs to. The request shows up as a trace.

    If nothing arrives, check the gateway’s logs: docker compose logs app in Standalone, docker compose logs agentcc-gateway in Distributed, or kubectl logs deployment/<fullname>-agentcc-gateway with Helm. A refused export never fails the request, so the logs are where it shows:

    • otlp exporter: collector rejected the credentials, dropping spans: the key pair is wrong, expired or disabled
    • otlp exporter: collector rejected batch, dropping spans: another refusal, such as a missing project_name
    • otlp exporter: max retries exceeded, dropping spans: the gateway can’t reach the endpoint, or the collector kept answering with a 5xx or 429 (the line’s reason says which)

Warning

In Standalone the gateway runs in the same app container as the API, the UI and the collector. If the gateway refuses to start (an unset variable or an invalid value), the whole container stops and restarts in a loop, and the UI goes down with it. docker compose logs app shows a failed to load config line naming the otel setting. Fix or remove the block, then run docker compose up -d again.


Metrics

Agent Command Center exports Prometheus metrics on the /-/metrics endpoint.

Available metrics

MetricTypeDescription
agentcc_requests_totalCounterTotal requests by model, provider, status code
agentcc_request_duration_secondsHistogramRequest latency distribution
agentcc_tokens_totalCounterTotal tokens (input + output) by model
agentcc_cost_totalCounterTotal cost in USD by model and provider
agentcc_cache_hits_totalCounterCache hits by strategy (exact/semantic)
agentcc_cache_misses_totalCounterCache misses
agentcc_provider_errors_totalCounterProvider errors by provider and error code
agentcc_circuit_breaker_stateGaugeCircuit breaker state (0=closed, 1=open, 2=half-open)
agentcc_rate_limit_exceeded_totalCounterRate limit rejections by key
agentcc_guardrail_triggered_totalCounterGuardrail triggers by guardrail name and action

Scrape configuration

# prometheus.yml
scrape_configs:
  - job_name: "agentcc-gateway"
    scrape_interval: 15s
    metrics_path: "/-/metrics"
    static_configs:
      - targets: ["agentcc-gateway:8080"]

Self-hosted metrics config

telemetry:
  metrics:
    enabled: true
    prometheus:
      enabled: true
      path: "/-/metrics"

Session tracking

Group related requests into sessions for conversation-level analytics. Set x-agentcc-session-id on each request in a conversation:

session_id = "user-123-conversation-456"
messages = []

# Each turn in the conversation shares the same session_id
messages.append({"role": "user", "content": "What's the capital of France?"})
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    session_id=session_id,
    user_id="user-123",
)
messages.append({"role": "assistant", "content": response.choices[0].message.content})

messages.append({"role": "user", "content": "What's its population?"})
response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    session_id=session_id,
    user_id="user-123",
)

Sessions appear in the dashboard under Agent Command Center > Sessions and show:

  • Total requests in the session
  • Cumulative cost
  • Models and providers used
  • Timeline of requests

Alerting

Configure alerts to get notified about issues. See Cost tracking > Budget alerts for alert configuration.

EventWhen it fires
Budget threshold crossedSpend exceeds configured percentage
Error rate spikeError rate exceeds threshold over a time window
Latency spikeP95 latency exceeds threshold
Guardrail triggeredA guardrail blocks or flags a request

Next Steps

Was this page helpful?

Questions & Discussion