Request Logging and Observability in Agent Command Center
Request logs, Prometheus metrics, and OpenTelemetry trace export for the gateway
About
Agent Command Center logs every request and response, exports metrics to Prometheus, can export OpenTelemetry traces from self-hosted gateways, and propagates trace IDs for distributed tracing. Request logging is on by default, with no extra setup. OpenTelemetry export is off until you turn it on.
Request logging
Every request through Agent Command Center is logged with:
- Request ID, trace ID, session ID
- Model requested and model actually used
- Provider that handled the request
- Input/output token counts
- Cost
- Latency
- Cache status (hit/miss/skip)
- Guardrail results
- Any errors or fallback events
Logs sync to the Future AGI dashboard automatically. View them in Agent Command Center > Logs.
Log levels
| Level | What’s logged |
|---|---|
error | Failed requests, provider errors, guardrail blocks |
warn | Fallbacks, slow requests, budget warnings |
info | Every request (default) |
debug | Full request/response bodies, header details |
For self-hosted deployments, set the log level in config.yaml:
logging:
level: info
Distributed tracing
Agent Command Center propagates trace IDs across the request lifecycle. Set x-agentcc-trace-id on incoming requests and the same ID appears in all downstream provider calls and logs.
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
trace_id="trace-from-my-app-abc123",
user_id="user-42",
)
print(response.agentcc.trace_id) # trace-from-my-app-abc123
print(response.agentcc.provider) # openai
print(response.agentcc.latency_ms) # 342
print(response.agentcc.cost) # 0.00015 raw = client.chat.completions.with_raw_response.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={
"x-agentcc-trace-id": "trace-from-my-app-abc123",
"x-agentcc-user-id": "user-42",
},
)
print(raw.headers.get("x-agentcc-trace-id"))
print(raw.headers.get("x-agentcc-cost")) curl -i https://gateway.futureagi.com/v1/chat/completions \
-H "Authorization: Bearer sk-agentcc-your-key" \
-H "x-agentcc-trace-id: trace-from-my-app-abc123" \
-H "x-agentcc-user-id: user-42" \
-H "Content-Type: application/json" \
-d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
# Look for x-agentcc-trace-id in response headers If you don’t set a trace ID, Agent Command Center generates one automatically. Use it for correlating gateway logs with your application logs.
OpenTelemetry export
Self-hosted gateways can also emit one OpenTelemetry span per request to any OTLP-compatible backend. Span attributes follow the gen_ai.* semantic conventions, and each span carries the gateway’s trace ID as agentcc.trace_id, so you can match it to its row in the request log above. The otel block in the gateway’s config.yaml turns export on; it’s off by default. Export never waits on the collector, so a slow or unreachable one doesn’t slow requests down.
Sending to your own Future AGI install? Go to Send gateway spans to self-hosted Observe.
otel:
enabled: true
exporter: otlp
endpoint: "http://otel-collector:4318" # /v1/traces is appended when the URL has no path
protocol: "http/protobuf"
service_name: "agentcc-gateway"
headers: # sent on every export request
Authorization: "${OTEL_AUTH_HEADER}" # for example "Bearer <token>"
Warning
The gateway speaks OTLP over HTTP with protobuf only, so point it at your collector’s OTLP/HTTP port (usually 4318), not the gRPC port 4317. Setting enabled: true without exporter: otlp prints spans to stdout instead of exporting them.
With enabled: true and exporter: otlp, the gateway refuses to start when:
endpointis missing, or isn’t an absolutehttp(s)URL- a header value is empty (usually a
${VAR}that isn’t set) protocolisgrpcorhttp/json
All otel keys:
| Key | Default | Description |
|---|---|---|
enabled | false | Turns span creation on |
exporter | stdout | stdout or otlp |
endpoint | none | Collector base URL; required for otlp |
protocol | http/protobuf | The only supported value |
headers | none | Headers sent with every export, for collector authentication. Use ${VAR} to keep credentials out of the file |
service_name | agentcc-gateway | service.name resource attribute |
sample_rate | 1.0 | Fraction of requests that get a span. 0 or unset is treated as 1.0; to stop spans, set enabled: false |
attributes | none | Extra OTLP resource attributes |
include_bodies | false | Attach the prompt and completion (input.value / output.value), redacted with the organization’s privacy settings like the request log |
trace_propagation | false | Honor an inbound W3C traceparent so the gateway span nests under the caller’s span |
metadata_attributes | true | Also emit each caller metadata key as agentcc.metadata.<key>, next to the metadata JSON object |
body_attributes | true | Emit unknown scalar request-body fields as agentcc.body.<key> attributes |
capture_headers | none | Allowlist of request headers to copy onto the span. Well-known credential headers (Authorization, Cookie, x-api-key and similar) are never copied |
Send gateway spans to self-hosted Observe
When you self-host Future AGI alongside the gateway, point the otel block at your install’s fi-collector, the service that receives traces for Observe.
Add the block to the gateway’s config file:
- Docker Compose: the file
AGENTCC_CONFIG_PATHin.envnames, relative to the repository root. If it’s still the bundledagentcc-gateway/config.example.yaml, copy it toagentcc-gateway/config.yamland setAGENTCC_CONFIG_PATH=agentcc-gateway/config.yamlfirst (see LLM gateway) - Helm: under
agentccGateway.config.otel, or in your own ConfigMap if you setagentccGateway.existingConfigMap
Create the API key and set its two variables (step 2) before you restart anything: the gateway won’t start while a ${VAR} in the block is unset.
otel:
enabled: true
exporter: otlp
endpoint: "http://127.0.0.1:4318" # see step 1 for your setup
protocol: "http/protobuf"
headers:
X-Api-Key: "${AGENTCC_OTEL_API_KEY}"
X-Secret-Key: "${AGENTCC_OTEL_SECRET_KEY}"
attributes:
project_name: "my-chatbot" # Observe project the spans land in
include_bodies: false # see step 4
trace_propagation: false # see step 5
-
Endpoint.
fi-collectortakes OTLP/HTTP on port 4318. Use the address the gateway can reach it on:Setup endpointStandalone ( docker-compose.yml)http://127.0.0.1:4318Distributed ( docker-compose.distributed.yml)http://fi-collector:4318Helm http://<fullname>-fi-collector:4318In Standalone the gateway runs in the same
appcontainer as the collector. With Helm,<fullname>isfutureagifor a release namedfutureagi, andkubectl get svcshows the exact Service name. -
API keys. The collector refuses an export without an API key pair. Create one in your own install (not app.futureagi.com) under Build > Keys, which Owners, Admins and workspace Admins can see. You get an API key and a secret key, and the secret key is shown only once, so copy both. Set them as
AGENTCC_OTEL_API_KEYandAGENTCC_OTEL_SECRET_KEY:- Docker Compose: in
.env - Helm: in a Secret listed under
agentccGateway.extraEnvFrom
Don’t reuse the names
FI_API_KEYandFI_SECRET_KEY. Every service in the install reads.env, and the API and the simulation runner read those two names too, so they’d start using your key. - Docker Compose: in
-
Project. Spans land in the Observe project named by the
project_nameresource attribute, in the workspace the API key belongs to. If no project has that name yet, it’s created on the first export. Pick a name no other project in that workspace uses: projects are matched by name, so an existing project with that name gets the spans. Without aproject_name, the collector refuses the export. -
Prompt and completion bodies.
include_bodiesis off by default, so spans carry metrics and metadata but no message content. Turning it on sends prompts and completions to Observe, redacted with your organization’s privacy settings, the same way the request log is. The request log has its own switch,logging.request_logging.include_bodies. -
Nesting under traceAI spans. If your application is instrumented with traceAI, sends its spans to the same
project_name, and passes a W3Ctraceparentheader on its calls to the gateway, settrace_propagation: true. The gateway span then nests under your application’s span instead of starting a second trace. Leave it off otherwise: any OpenTelemetry instrumentation in your application can send atraceparent, and the gateway span would then point at a parent Observe never receives, so the trace isn’t listed. Only atraceparentmarked as sampled is honored. -
Apply and check. The gateway reads its config once, at startup, so restart it after every change:
- Docker Compose, first time or after editing
.env:docker compose up -d(docker compose restartdoesn’t re-read.env) - Docker Compose, after editing only the config file:
docker compose restart appin Standalone, ordocker compose restart agentcc-gatewayin Distributed (docker compose up -ddoesn’t pick up a config-file change) - Helm:
helm upgrade. If you useexistingConfigMap, or changed only the Secret, also runkubectl rollout restart deployment/<fullname>-agentcc-gateway
Send one request through the gateway, then open the
my-chatbotproject in Observe, in the workspace the API key belongs to. The request shows up as a trace.If nothing arrives, check the gateway’s logs:
docker compose logs appin Standalone,docker compose logs agentcc-gatewayin Distributed, orkubectl logs deployment/<fullname>-agentcc-gatewaywith Helm. A refused export never fails the request, so the logs are where it shows:otlp exporter: collector rejected the credentials, dropping spans: the key pair is wrong, expired or disabledotlp exporter: collector rejected batch, dropping spans: another refusal, such as a missingproject_nameotlp exporter: max retries exceeded, dropping spans: the gateway can’t reach the endpoint, or the collector kept answering with a 5xx or 429 (the line’sreasonsays which)
- Docker Compose, first time or after editing
Warning
In Standalone the gateway runs in the same app container as the API, the UI and the collector. If the gateway refuses to start (an unset variable or an invalid value), the whole container stops and restarts in a loop, and the UI goes down with it. docker compose logs app shows a failed to load config line naming the otel setting. Fix or remove the block, then run docker compose up -d again.
Metrics
Agent Command Center exports Prometheus metrics on the /-/metrics endpoint.
Available metrics
| Metric | Type | Description |
|---|---|---|
agentcc_requests_total | Counter | Total requests by model, provider, status code |
agentcc_request_duration_seconds | Histogram | Request latency distribution |
agentcc_tokens_total | Counter | Total tokens (input + output) by model |
agentcc_cost_total | Counter | Total cost in USD by model and provider |
agentcc_cache_hits_total | Counter | Cache hits by strategy (exact/semantic) |
agentcc_cache_misses_total | Counter | Cache misses |
agentcc_provider_errors_total | Counter | Provider errors by provider and error code |
agentcc_circuit_breaker_state | Gauge | Circuit breaker state (0=closed, 1=open, 2=half-open) |
agentcc_rate_limit_exceeded_total | Counter | Rate limit rejections by key |
agentcc_guardrail_triggered_total | Counter | Guardrail triggers by guardrail name and action |
Scrape configuration
# prometheus.yml
scrape_configs:
- job_name: "agentcc-gateway"
scrape_interval: 15s
metrics_path: "/-/metrics"
static_configs:
- targets: ["agentcc-gateway:8080"]
Self-hosted metrics config
telemetry:
metrics:
enabled: true
prometheus:
enabled: true
path: "/-/metrics"
Session tracking
Group related requests into sessions for conversation-level analytics. Set x-agentcc-session-id on each request in a conversation:
session_id = "user-123-conversation-456"
messages = []
# Each turn in the conversation shares the same session_id
messages.append({"role": "user", "content": "What's the capital of France?"})
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
session_id=session_id,
user_id="user-123",
)
messages.append({"role": "assistant", "content": response.choices[0].message.content})
messages.append({"role": "user", "content": "What's its population?"})
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
session_id=session_id,
user_id="user-123",
)
Sessions appear in the dashboard under Agent Command Center > Sessions and show:
- Total requests in the session
- Cumulative cost
- Models and providers used
- Timeline of requests
Alerting
Configure alerts to get notified about issues. See Cost tracking > Budget alerts for alert configuration.
| Event | When it fires |
|---|---|
| Budget threshold crossed | Spend exceeds configured percentage |
| Error rate spike | Error rate exceeds threshold over a time window |
| Latency spike | P95 latency exceeds threshold |
| Guardrail triggered | A guardrail blocks or flags a request |
Next Steps
Questions & Discussion