> For the complete documentation index, see [llms.txt](https://docs.observal.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.observal.io/self-hosting/backup-and-restore.md).

# Backup and restore

What to back up, how often, and how to restore. Observal has three persistent stores with very different loss profiles.

## What matters, in order

| Priority      | Volume        | Contents                                               | Loss impact                                                           |
| ------------- | ------------- | ------------------------------------------------------ | --------------------------------------------------------------------- |
| **Critical**  | `apidata`     | JWT signing keys                                       | Every session invalidated. No recovery without the keys.              |
| **Critical**  | `pgdata`      | Users, RBAC, registry metadata, agents                 | All accounts and registry lost.                                       |
| **Important** | `chdata`      | Session events, aggregates, audit, and security events | All telemetry lost. Accounts and registry survive.                    |
| **Low**       | `grafanadata` | Custom Grafana dashboards                              | Custom dashboards lost; provisioned defaults come back automatically. |
| **Low**       | `redisdata`   | Job queue state                                        | In-flight jobs lost; they re-queue on next worker restart.            |

**Always back up `apidata` and `pgdata` together.** Backing up one without the other leaves you in a broken state after restore.

## Backup cadence

| Cadence              | What                | Retention                                  |
| -------------------- | ------------------- | ------------------------------------------ |
| Daily                | `pgdata`, `apidata` | 30 days                                    |
| Weekly               | `chdata`            | 12 weeks                                   |
| Before every upgrade | All three           | Keep until the upgrade is confirmed stable |

## Postgres backup

Use `pg_dump` inside the running container:

```bash
docker compose -f docker/docker-compose.yml exec -T observal-db \
  pg_dump -U postgres observal | gzip > observal-pg-$(date +%Y%m%d).sql.gz
```

Restore into a fresh DB:

```bash
docker compose -f docker/docker-compose.yml down -v
docker compose -f docker/docker-compose.yml up -d observal-db

zcat observal-pg-20260421.sql.gz | \
  docker compose -f docker/docker-compose.yml exec -T observal-db \
  psql -U postgres observal

docker compose -f docker/docker-compose.yml up -d
```

## JWT key backup (`apidata` volume)

The API container mounts the `apidata` volume at `/data` and stores keys at `/data/keys`. Tar it out:

```bash
docker compose -f docker/docker-compose.yml exec -T observal-api \
  tar czf - -C /data keys > observal-keys-$(date +%Y%m%d).tar.gz
```

Restore:

```bash
# With the API stopped
docker compose -f docker/docker-compose.yml stop observal-api

docker run --rm \
  -v observal_apidata:/data \
  -v "$(pwd)":/backup \
  alpine sh -c "cd /data && tar xzf /backup/observal-keys-20260421.tar.gz"

docker compose -f docker/docker-compose.yml start observal-api
```

## ClickHouse backup

### Option A - volume snapshot (simplest)

```bash
docker compose -f docker/docker-compose.yml stop observal-clickhouse
docker run --rm -v observal_chdata:/data -v "$(pwd)":/backup \
  alpine tar czf /backup/observal-ch-$(date +%Y%m%d).tar.gz -C /data .
docker compose -f docker/docker-compose.yml start observal-clickhouse
```

Downtime: however long the tar takes (a minute to tens of minutes depending on size).

### Option B - ClickHouse native `BACKUP` (no downtime)

```bash
docker compose -f docker/docker-compose.yml exec observal-clickhouse \
  clickhouse-client --query "BACKUP DATABASE observal TO Disk('backups', 'observal-$(date +%Y%m%d).zip')"
```

Requires configuring a backup disk in ClickHouse config; see [ClickHouse docs](https://clickhouse.com/docs/en/operations/backup).

Restore:

```bash
docker compose -f docker/docker-compose.yml exec observal-clickhouse \
  clickhouse-client --query "RESTORE DATABASE observal FROM Disk('backups', 'observal-20260421.zip')"
```

## Restore order

If you're restoring from backup after a catastrophic failure:

1. Stop the whole stack: `docker compose down`.
2. Restore `apidata` (JWT keys) first.
3. Restore `pgdata` (Postgres).
4. Restore `chdata` (ClickHouse).
5. Bring up the stack: `docker compose up -d`.
6. Smoke test: `observal auth login`, `observal auth status`.

Skipping step 2 works but every user has to re-login.

## Verifying a backup

Test restores in a staging environment at least quarterly. Untested backups are guesses.

Smoke test after restore:

```bash
observal auth login
observal auth whoami              # you should be your pre-backup user
observal agent list               # registry should be intact
observal ops traces --limit 5     # traces up to the backup timestamp should be visible
```

## Automated backup

A minimal cron setup (on the Docker host):

```cron
# Daily at 03:00 - Postgres + JWT keys
0 3 * * * cd /opt/Observal && \
  docker compose -f docker/docker-compose.yml exec -T observal-db \
  pg_dump -U postgres observal | gzip > /backups/pg-$(date +\%Y\%m\%d).sql.gz

0 3 * * * cd /opt/Observal && \
  docker compose -f docker/docker-compose.yml exec -T observal-api \
  tar czf - -C /data keys > /backups/keys-$(date +\%Y\%m\%d).tar.gz

# Weekly Sunday at 04:00 - ClickHouse
0 4 * * 0 cd /opt/Observal && \
  docker compose -f docker/docker-compose.yml stop observal-clickhouse && \
  docker run --rm -v observal_chdata:/data -v /backups:/backup alpine \
    tar czf /backup/ch-$(date +\%Y\%m\%d).tar.gz -C /data . && \
  docker compose -f docker/docker-compose.yml start observal-clickhouse
```

Ship the `/backups` directory offsite (S3, B2, rsync to another host).

## Next

→ [Troubleshooting](/self-hosting/troubleshooting.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.observal.io/self-hosting/backup-and-restore.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.