Monitoring

These are the signals a self-hosted instance exposes: container health, the backend’s health endpoint, the collector’s, the first-run setup checks, the LLM gateway’s metrics and, on Distributed, the PeerDB console. This page covers each, what a bad reading looks like, and where to go when you get one.

With the production overlay, add its --env-file and -f flags to every docker compose command on this page, as in Start Distributed with the production overlay.

Container health

Every long-running Future AGI image declares a Docker health check, as the data stores do, so docker compose ps shows healthy or unhealthy for the application too:

docker compose ps        # STATUS shows healthy / unhealthy per service
docker stats             # live CPU and memory per container

• Standalone: the app container’s check probes the API (/health/), the collector, the LLM gateway, the UI and object storage inside the container, every 15 seconds. Its start period is 3600 seconds, because a first boot on a slow host can migrate for over half an hour; a passing probe marks it healthy at once. docker compose logs -f app starts with a summary of the configuration (setup, version, URLs, which integrations are on, telemetry) and ends with Future AGI is ready in .... A failed start-up step prints its error there and retries after 30 seconds.
• Distributed: backend, frontend, fi-collector, agentcc-gateway, serving and code-executor probe their own endpoints. The workers report healthy without probing anything, so read their logs (docker compose logs worker), and the one-shot setup jobs should have exited with code 0 (docker compose ps -a).

A container cycling through Restarting in docker compose ps often means it ran out of memory. Size the Docker VM as Requirements says. On Distributed, clickhouse and worker are the containers to watch.

Backend health

The backend serves an unauthenticated health check at /health/. It’s the endpoint ./bin/install polls while it waits for the stack, so it tells you the application layer is serving rather than merely running:

curl -sf http://localhost:8000/health/

It answers 503 while the API’s embedded Temporal worker is down, so on Standalone it covers the worker too. During a Standalone first boot the API answers 503 with {"status": "starting", "phase": "..."} until the start-up steps finish.

Collector health

On Distributed, the collector serves /healthz on its admin port, 127.0.0.1:9464 (FI_COLLECTOR_ADMIN_PORT):

curl -s http://localhost:9464/healthz

Once the collector has sent more than 100 insert batches since it started, it answers 503 while more than half of them have failed. Its JSON body carries the insert counters. Alert on it rather than just checking it: a 503 means spans are failing to reach ClickHouse and landing in the collector’s dead-letter file (in futureagi_fi-collector-data) instead of being queryable.

Standalone does not publish the admin port: the app container’s health check covers the collector, and its dead-letter file sits in futureagi_app-data. See Backups & restore.

Setup checks

The first-run setup screen checks every service the app depends on (databases, cache, object storage, gateway, task engine, trace ingestion, sandbox, TLS) and gives the fix for the setup you run. The same checks answer as JSON, without authentication, on the API. The overall result is .result.status: ok, or issues when a required check failed. The top-level status is true on every 200 answer, whatever the checks found.

curl -s http://localhost:8000/api/setup-checks/ | jq -r .result.status   # ok, or issues

On an install with an Enterprise Edition license (EE_LICENSE_KEY), the endpoint answers 404.

LLM gateway metrics

The LLM gateway serves Prometheus metrics at /-/metrics when prometheus.enabled is true in its config file: the file AGENTCC_CONFIG_PATH names, a copy of agentcc-gateway/config.example.yaml as for adding providers (Helm: agentccGateway.config.prometheus.enabled). The endpoint needs the gateway’s admin token:

TOKEN=$(docker compose exec -T app printenv AGENTCC_ADMIN_TOKEN)   # Distributed: backend instead of app
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8090/-/metrics

After editing the config file, run docker compose restart app (Distributed: agentcc-gateway). The API and the collector serve no Prometheus endpoint; watch the data stores with their own exporters.

PeerDB replication

On Distributed with the peerdb or all profile, the Postgres-to-ClickHouse pipeline has its own console at localhost:3001. Mirror status there tells you whether trace analytics are keeping up with Postgres. A mirror in anything other than running means the dashboard is reading stale data. Re-running PeerDB init does not restart or resync a mirror: see Analytics data is stale. Standalone and Helm sync through an outbox inside the app and have no PeerDB.

Helm

On Kubernetes, helm test checks that the API, the UI, the collector and the gateway answer, and the setup screen shows kubectl fixes. The chart has no ServiceMonitor yet. See Helm (Kubernetes).

Dive deeper