Monitoring

Four things are worth watching on a self-hosted instance: container health, the backend’s health endpoint, the collector’s, and the PeerDB replication console. This page covers each, what a bad reading looks like, and where to go when you get one.

Container health

The data stores (Postgres, ClickHouse, Redis, RabbitMQ, MinIO, Temporal) ship Docker health checks. The application services declare none, so they only ever read running, which is why the two endpoints below matter. Either way, docker compose ps is the fastest read on what’s up:

docker compose ps        # STATUS shows healthy / unhealthy per service
docker stats             # live CPU and memory per container

Watch memory on clickhouse and the Temporal worker first. They’re the resource drivers, each sitting around 1 GB at steady state, and an OOM there is the most common cause of a stall. The tell is a container cycling through Restarting in docker compose ps: raise the host’s memory to the requirements floor if you see it.

Backend health

The backend serves an unauthenticated health check at /health/. It’s the same endpoint ./bin/install polls while it waits for the stack, so it’s the one signal that tells you the application layer is actually serving rather than merely running:

curl -sf http://localhost:8000/health/

fi-collector health

The fi-collector exposes an admin endpoint on 127.0.0.1:9464 (FI_COLLECTOR_ADMIN_PORT):

curl -s http://localhost:9464/healthz

This one is worth alerting on rather than just checking. It returns 200 unless the collector’s dead-letter rate crosses its threshold, so a non-200 means spans are failing to reach ClickHouse and piling up in futureagi_fi-collector-data instead of being queryable.

PeerDB replication

The Postgres-to-ClickHouse pipeline has its own console at localhost:3001. Mirror status there tells you whether trace analytics are keeping up with Postgres. A mirror in anything other than running means the dashboard is reading stale. Re-running init usually clears it, and Troubleshooting has the command and the first-boot race that causes most of these.

Dive deeper