Monitoring

The stack has no Prometheus /metrics endpoint yet (the fi-collector lists a metrics exporter as a TODO), so monitoring today is built from what the containers already expose: their Docker health checks, the fi-collector’s admin health endpoint, and the PeerDB console. This page covers those and the signals worth watching.

Container health

The data stores (Postgres, ClickHouse, Redis, RabbitMQ, MinIO, Temporal) ship Docker health checks; the application services just show running. Either way, docker compose ps is the fastest read on what’s up:

docker compose ps        # STATUS shows healthy / unhealthy per service
docker stats             # live CPU and memory per container

Watch memory on clickhouse and the Temporal worker first. They’re the resource drivers, and an OOM there is the most common cause of a stall.

fi-collector health

The fi-collector exposes an admin endpoint on 127.0.0.1:9464 (FI_COLLECTOR_ADMIN_PORT), which serves a health check. Hit it to confirm the collector is up:

curl -s http://localhost:9464/healthz

PeerDB replication

The Postgres-to-ClickHouse pipeline has its own console at localhost:3001. Mirror status there tells you whether trace analytics are keeping up with Postgres. A mirror in anything other than running means the dashboard is reading stale.

Dive deeper