Upgrades & rollback

Each service runs a published image, tagged by version variables in .env. Two exceptions build locally: ./bin/install --from-source builds the images from your checkout and records FUTURE_AGI_VERSION=local, and Distributed builds the trace collector from ./fi-collector until FI_COLLECTOR_VERSION names a published tag. So an upgrade pulls new images, and a rollback goes back to the previous release’s files and images, plus a database restore when a migration ran in between. This page covers Standalone and Distributed; on Kubernetes, see Upgrade and roll back for the Helm chart.

Upgrade to a new release

Take a backup of Postgres and ClickHouse before you start. Migrations run on start and a rollback does not reverse them, so the backup is the only way back from one that goes wrong. Backups & restore has the commands.

git pull
./bin/install            # add --from-source if you run a branch from source

git pull
docker compose pull
docker compose up -d     # Distributed with a locally built collector: add --build

docker compose logs -f app    # ends with "Future AGI is ready in ..."

docker compose ps -a                          # the setup jobs should show Exited (0)
docker compose logs postgres-schema-bootstrap # the migration output, and the error if one failed

docker compose run --rm peerdb-init

Upgrade with the production overlay

The overlay’s start-up jobs only check the schema and the mirrors, so an upgrade applies them by hand first. Every step uses deploy/.env.production and the overlay’s command, as deploy/README.md › Upgrades describes:

1. Back up Postgres and ClickHouse, and deploy/.env.production.

2. Get the release’s files: git fetch --tags && git checkout vX.Y.Z.

3. Set every version variable in deploy/.env.production to the new release. On a GPU host, SERVING_VERSION takes the -gpu tag.

4. Pull the images:

   docker compose --env-file deploy/.env.production \
     -f docker-compose.distributed.yml -f deploy/docker-compose.production.yml pull

5. Re-run the initialization jobs from deploy/README.md with the same commands as on the first install. Each one changes nothing that is already current, so a stalled upgrade resumes from the job that failed.

6. Start the stack:

   docker compose --env-file deploy/.env.production \
     -f docker-compose.distributed.yml -f deploy/docker-compose.production.yml up -d --no-build --wait --wait-timeout 1200

Spans stored before the collector indexed their attributes stay out of attribute suggestions until you run the span backfill: see Upgrading an existing install: backfill before you cut over.

To roll back an overlay install, go back only to a predecessor you have rehearsed, with its exact images and configuration. Changing the tags alone does not undo schema or mirror changes.

Roll back a bad release

Roll back the files and the images together. The old images on the new compose files work only while the two releases share the compose layout, which changed when Standalone became the default: to roll back across that change, see Roll back to a release before Standalone became the default.

First find the tags you’re on:

docker compose images    # IMAGE and TAG per running service

Check out the release you’re going back to (release tags are vX.Y.Z, matching the repository’s git tags):

git checkout vX.Y.Z

Then set each version variable back to it in .env:

FUTURE_AGI_VERSION=vX.Y.Z
SERVING_VERSION=vX.Y.Z          # only with the ml profile; vX.Y.Z-gpu on GPU hosts
CODE_EXECUTOR_VERSION=vX.Y.Z    # only with the sandbox profile

FUTURE_AGI_VERSION=vX.Y.Z       # backend and workers
FRONTEND_VERSION=vX.Y.Z
AGENTCC_GATEWAY_VERSION=vX.Y.Z
FI_COLLECTOR_VERSION=vX.Y.Z
SERVING_VERSION=vX.Y.Z          # vX.Y.Z-gpu on GPU hosts
CODE_EXECUTOR_VERSION=vX.Y.Z
SIMULATION_RUNNER_VERSION=vX.Y.Z

docker compose pull
docker compose up -d

Downtime is about the time the app (or the backend) takes to restart. A variable left empty means latest, so for anything carrying real traffic, pin every image your setup runs rather than floating.

To pin an exact image digest, name it in a docker-compose.override.yml (image: futureagi/standalone:vX.Y.Z@sha256:<digest>), never through FUTURE_AGI_VERSION, which is also reported as the version in telemetry. Verifying an image shows how to read the digest and trace an image to its commit. On Distributed, list the override in COMPOSE_FILE in .env: see Move to managed data stores.

Roll back to a release before Standalone became the default

The releases before Standalone became the default run RabbitMQ, which the current compose files no longer define, and their docker-compose.yml is the Distributed stack. Rolling back to one takes that release’s files, its images and the backups you took before the upgrade:

1. Stop the upgraded stack. This removes the containers and keeps the volumes, rabbitmq-data among them:

   docker compose down

2. Check out the release you ran before the upgrade: git checkout vX.Y.Z.

3. In .env, delete the COMPOSE_FILE line: that release’s docker-compose.yml is already Distributed. Set every version variable to its tag, since an empty one means latest, which is no longer that release. The exception is FI_COLLECTOR_VERSION: delete it, and that release builds the collector from its own source.

4. Restore Postgres and ClickHouse from the backups you took before the upgrade, as the first three steps of Restore describe.

5. Start that release, which recreates RabbitMQ from its volume:

   docker compose up -d --build    # --build builds the collector from that release's source

Upgrading older installs

An install made before Standalone became the default runs the Distributed stack from docker-compose.yml. That file is now Standalone, and Distributed lives in docker-compose.distributed.yml. Such an install stays on Distributed: moving its data to Standalone is not supported.

Your install is one if its checkout, before you pull, has no docker-compose.distributed.yml, or if docker volume ls lists a rabbitmq-data volume for its Compose project (futureagi_rabbitmq-data by default), which only releases before Standalone became the default create.

Keep it on Distributed

git pull && ./bin/install recognises the old install by its volumes or containers, keeps it on Distributed, and records COMPOSE_FILE=docker-compose.distributed.yml in .env at once, saying so. Deleting that line does not move the install: the next run puts it back.

If you upgrade with plain docker compose instead, add the line to .env before you run docker compose up -d:

echo "COMPOSE_FILE=docker-compose.distributed.yml" >> .env   # with an override: docker-compose.distributed.yml:docker-compose.override.yml

Without it, Compose reads the new default file under the same project name. It recreates your postgres and clickhouse containers with Standalone’s small-host settings while the old backend and workers keep running against them, and adds an app container that fails on the ports they hold. Your data stays in its volumes. To recover, add the line, remove the stray container, and re-run the installer (or docker compose up -d), which restores Distributed’s settings:

echo "COMPOSE_FILE=docker-compose.distributed.yml" >> .env
docker compose -f docker-compose.yml rm -sf app
./bin/install

If you run such an install with the production overlay, the command in your scripts and service units names -f docker-compose.yml. Replace it with -f docker-compose.distributed.yml before you run anything after git pull: the old command now layers the overlay on the Standalone file, under the same project name. Add AGENTCC_WEBHOOK_SECRET (openssl rand -hex 32) to deploy/.env.production, since the installer, which fills it elsewhere, does not run there; without it the app refuses the gateway’s request logs. Then follow Upgrade with the production overlay.

Remove the retired services

Distributed no longer runs RabbitMQ (Redis carries live updates), and RABBITMQ_USER and RABBITMQ_PASSWORD are no longer read. Nor does it run the Kafka catalog sequencer (fi-property-catalog-sequencer, with its property-catalog-supervisor and property-catalog-postgres-bootstrap jobs) that some older releases ran. An upgraded install keeps whichever of these containers it has until you remove them. Once the upgraded stack is healthy:

docker compose up -d --remove-orphans      # removes the retired containers

Their volumes, rabbitmq-data and, where the sequencer ran, property-catalog-sequencer-data, are unused from then on, and docker compose down -v does not remove them either. Rolling back to a release before Standalone became the default needs them, so keep them until the upgrade has proven itself and you will not roll back, then delete the ones docker volume ls lists (the prefix is your Compose project name):

docker volume rm futureagi_rabbitmq-data futureagi_property-catalog-sequencer-data

Check .env

• Password-reset links: the old .env.example shipped OSS_RETURN_PASSWORD_RESET_LINK=true, which hands a reset link to anyone who asks. Unless you want that, delete the line and run docker compose up -d. ./bin/install warns while it is there.
• Secrets: the installer never changes an existing secret. It warns when one still uses a default published in the repository, and fills only INTEGRATION_ENCRYPTION_KEY and AGENTCC_WEBHOOK_SECRET when they are unset. Replace the defaults as the Checklist describes.
• ClickHouse password: CH_PASSWORD stays empty, so ClickHouse keeps running without a password until you set one. How to add one, and why on Distributed that also means updating PeerDB’s ch_dest peer, is in Secrets the installer generates.

Attribute suggestions for older traces

Trace filters, dashboard widgets and task filters suggest the span attributes and values the collector indexed as spans arrived. Spans stored before the collector indexed their attributes are missing from the suggestions until you backfill them, although filtering on them still works when you type the name and value. See Filters suggest no attributes or values for older traces; for Distributed, the installer points to fi-collector/PROPERTY_CATALOG_OSS.md.

LLM gateway changes

• Private provider URLs: a provider an organization adds with a base URL on a private network needs AGENTCC_ALLOW_PRIVATE_PROVIDER_URLS=true. That now includes CGNAT addresses (100.64.0.0/10), such as a Tailscale IP, which older gateways let through. See Private and local provider URLs.
• Organization base URLs: a provider an organization adds with its own base URL, on a provider ID that also exists in the gateway’s config.yaml, now really goes to that URL. Before, the organization’s key went to the config.yaml base URL. A private URL there needs the setting above.
• Operator credentials: an organization’s Bedrock or Vertex provider no longer borrows the AWS or Google credentials from the operator’s config.yaml: it authenticates with its own.
• Key IDs: the gateway now issues random key IDs. An older gateway numbers them (key_1, key_2, …) from a counter that starts over when it restarts, and a newer API refuses a reissued ID instead of overwriting the older key: The gateway issued key id '...', which already belongs to another API key; the new key was discarded. Upgrade the gateway so it issues unique key ids, then try again. On Distributed, upgrade the gateway image together with the backend; Standalone ships both in one image.
• Log lines: if you alert on gateway logs, two request-log lines are gone: log flusher: webhook returned server error, re-enqueuing records and log flusher: max retries exceeded after server errors, dropping records. Server errors now log log flusher: send failed, re-enqueuing records and log flusher: max retries exceeded, dropping records, with a status attribute. On shutdown the gateway now sends its buffered request logs first and logs log flusher: delivered buffered request logs before shutdown, or log flusher: request logs not delivered before shutdown when it could not.

Switch setups

Moving an install between Standalone, Distributed and Helm with its data is not supported: see Switch setups.

Dive deeper