Skip to main content
Safe upgrade flow for the Observal server stack.
[!IMPORTANT] When upgrading into the release that introduces complete-folder skill delivery, follow the skill-folder rollout procedure instead of using the quick, standard, or blue/green instructions below unchanged. Those paths do not guarantee that old API and worker processes have stopped before the new init container runs PostgreSQL migrations. Keep folder delivery off, back up and rehearse the historical upgrade, and schedule a maintenance window to drain old processes before migration; enable delivery only after a homogeneous rollout and separate approval.
If you installed Observal via observal server start (the embedded stack), the CLI handles upgrades automatically:
This pulls new Docker images, backs up PostgreSQL, recreates containers, and runs health checks. If the health check fails, it requests the previous image version again. Local shell and Docker access authorize the operation; the command does not require a reachable API or API role. See observal server upgrade for full details.

Server-package upgrades

If you installed with install-server.sh, back up .env, secrets/, and the apidata and pgdata volumes before rerunning the installer for the target release. Setup detects the existing .env. Choose the default No response to preserve custom values and existing direct credentials. This only stages the new package and retains a backup: no containers restart and no migrations run. An exit code of zero means staging succeeded, not that the server has been upgraded. The installer records the prior bind address when an older install has no OBSERVAL_BIND_ADDRESS setting. For the complete-folder release, follow the supervised rollout: keep the delivery gate off, stop all old API and worker processes before starting the new init container, then run Compose from the installed package with the same optional observability overlay/profile used by the old deployment. For a core-only package installation, after staging the package and verifying backups and the gate:
Add -f docker-compose.observability.yml and --profile grafana if those were used previously; do not silently drop optional services. Verify the API externally through the load balancer, not only via its container health check. Choosing Yes at the configuration prompt instead intentionally rebuilds the general configuration from the new template and starts services; for this release, stop old API and worker processes before running the installer, because setup will start the new init container. Core application, PostgreSQL, ClickHouse, Grafana, and demo credentials are preserved or migrated into restricted files, but unrelated custom environment entries must be reapplied.

Before a manual upgrade

  1. Back up pgdata and apidata. See Backup and restore. Backing up chdata is nice-to-have; losing telemetry is painful but not catastrophic.
  2. Read the CHANGELOG for the releases you’re jumping across. Note any breaking changes.
  3. Pin the version you’re upgrading to: don’t git pull main blindly. Check out a release tag or a known-good commit.

Standard upgrade

The init container applies pending Postgres Alembic migrations and ClickHouse SQL migrations before the API starts. Watch the init logs for migration output:

Zero-downtime upgrade (small teams)

If you run a single instance and have a ~30-second maintenance window:
  1. Back up pgdata, apidata, chdata.
  2. Stop the API and worker: docker compose stop observal-api observal-worker.
  3. Apply migrations out of band with alembic upgrade head and python -m services.clickhouse.migrations from observal-server, or run the init container once.
  4. Pull/rebuild new images: docker compose pull && docker compose build observal-api observal-worker.
  5. Start: docker compose up -d; wait for the new API to become healthy, then docker compose restart observal-lb so nginx resolves the new container IPs.
  6. Smoke test through the public load balancer: curl -fsS http://localhost/health, then observal auth status --output json && observal ops telemetry status --output json.
Web UI, Postgres, ClickHouse, Redis stay up throughout. Users see a brief API outage (~15–30 s).

Zero-downtime at scale

For blue/green upgrades on large deployments:
  1. Run a second stack (docker-compose.yml with different project name and host ports) behind a reverse proxy.
  2. Apply migrations before traffic cutover. Alembic handles Postgres and services.clickhouse.migrations handles ClickHouse. Additive migrations should work with API N-1 and N during the rollout.
  3. Bring up the green stack pointing at the same pgdata / chdata / apidata volumes.
  4. Flip the reverse proxy to green.
  5. Decommission blue.
If a migration is not additive (rare, but happens: column drops, type changes), it gets called out in the CHANGELOG and requires a brief outage. Plan the window.

Rolling back

If the new version breaks:
  1. docker compose -f docker/docker-compose.yml down
  2. git checkout <previous-version>
  3. docker compose -f docker/docker-compose.yml up --build -d
The catch: if the failing version already applied a migration, downgrading to the previous API version may leave you running against a schema it doesn’t know about. Options:
  • If the migration is additive (most are): the previous version works fine against the newer schema.
  • If the migration is destructive: restore from the pre-upgrade pgdata backup. This is why the backup is step 1.

CLI upgrades

CLI upgrades are independent of server upgrades. Users:
The CLI speaks a stable contract with the server. A newer CLI works against an older server and vice versa, within a release or two. On the first command after upgrading from a release that generated observal-shim entries, the CLI restores the original MCP command in known home and current-project config files. It writes a .pre-unshim.bak copy before each change. A malformed wrapped entry stops the command with a visible migration error rather than guessing at the original executable.

Zero-downtime for the web UI

The web UI is a static Vite build and restarts instantly. Users see a brief reload if they are on the page during the deploy. No special handling required.

Next

→ Backup and restore