Production

Hardening checklist, backups, monitoring, and upgrade runbook.

About

Run through this before exposing the stack to real users. Covers secrets, TLS, swapping in managed data stores, backup commands for Postgres/ClickHouse/MinIO, Prometheus monitoring, and the upgrade and rollback runbook.

Hardening checklist

Secrets — replace all CHANGEME values before going live:

openssl rand -hex 32    # SECRET_KEY, AGENTCC_INTERNAL_API_KEY
openssl rand -base64 24 # PG_PASSWORD, MINIO_ROOT_PASSWORD

Runtime flags in .env:

  • ENV_TYPE=prod
  • FAST_STARTUP=false
  • GRANIAN_WORKERS=<your CPU count>

TLS — the frontend and backend don’t terminate TLS. Put Caddy, nginx, or Traefik in front:

# Caddyfile (simplest — auto-issues Let's Encrypt certs)
app.yourcompany.com    { reverse_proxy localhost:3000 }
api.yourcompany.com    { reverse_proxy localhost:8000 }

After setting up TLS, set VITE_HOST_API=https://api.yourcompany.com in .env and rebuild:

docker compose build frontend && docker compose up -d frontend

Managed data stores — for production, replace compose-managed services:

ReplaceWithChange
postgresRDS / Aurora / Cloud SQLSet PG_* vars to managed endpoint
clickhouseClickHouse CloudSet CH_HOST, CH_PORT, etc.
redisElastiCache / UpstashSet REDIS_URL
minioAWS S3Set S3_ENDPOINT_URL=https://s3.amazonaws.com + AWS creds

Note

code-executor requires privileged: true. Run on EC2 / GCE instances — not Fargate or Cloud Run.

Secrets manager — use AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager instead of a plain .env file.


Backups

PostgreSQL

# Backup
docker compose exec postgres \
  pg_dump -U futureagi -d futureagi --format=custom \
  > backup-$(date +%F).dump

# Restore
docker compose exec -T postgres \
  pg_restore -U futureagi -d futureagi --clean --if-exists \
  < backup-2026-04-22.dump

Volumes: future-agi_postgres-data · future-agi_clickhouse-data · future-agi_redis-data · future-agi_minio-data · future-agi_peerdb-catalog-data · future-agi_peerdb-minio-data

ClickHouse

BACKUP TABLE default.traces TO S3('s3://your-bucket/ch-backup/', 'KEY', 'SECRET');

ClickHouse data can also be rebuilt from scratch by re-running PeerDB init since it replicates from Postgres.

MinIO

mc alias set local http://localhost:9005 futureagi <MINIO_ROOT_PASSWORD>
mc alias set s3 https://s3.amazonaws.com <AWS_KEY> <AWS_SECRET>
mc mirror local/ s3/your-bucket/

Monitoring

Backend exposes Prometheus metrics at http://localhost:8000/metrics. Add a scraper:

# prometheus.yml
scrape_configs:
  - job_name: futureagi
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: /metrics

Key signals: backend error rate, Temporal workflow success/failure, Postgres WAL lag (PeerDB health), ClickHouse query latency, PeerDB mirror status at localhost:3001.


Upgrades

git pull
docker compose build
docker compose up -d

Migrations run automatically. If a migration fails: docker compose exec backend python manage.py migrate

If release notes mention PeerDB changes: docker compose run --rm peerdb-init bash /setup.sh

Rollback:

git log --oneline -5
git checkout <previous-hash>
docker compose build && docker compose up -d

Next Steps

Was this page helpful?

Questions & Discussion