Skip to main content
Move PostgreSQL registry data and ClickHouse telemetry between Observal deployments. Migration uses the supplied database connections directly. Local shell and database access are the authorization boundary; the command does not authenticate against a configured Observal API. Install the optional dependency first:
Keep connection URLs in environment variables or secret files managed by the shell. Source commands read DATABASE_URL and CLICKHOUSE_URL; target commands read TARGET_DATABASE_URL and TARGET_CLICKHOUSE_URL. Explicit URL options remain available when no secret is embedded. Do not paste credentials into shared shell history, logs, or issue reports. JSON results and categorized errors never echo a connection URL.

​
Workflow

  1. Export PostgreSQL. This creates a checksummed registry archive and a migration manifest.
  2. Validate and import PostgreSQL on the target.
  3. Export ClickHouse using the PostgreSQL migration manifest.
  4. Validate and import ClickHouse on the target.
PostgreSQL must be imported first so referenced users and agents exist before telemetry validation.

​
PostgreSQL export

--file/-f selects the archive destination. --output/-o always selects table or json. Existing destinations fail with a conflict instead of being overwritten. The archive and sidecar manifest are written atomically with owner-only permissions. A partial archive is not published after failure. Example JSON fields:

​
PostgreSQL validation and import

Validation checks archive structure and SHA-256 checksums. When a target URL is provided, it also compares table row counts. Checksum failure returns a categorized validation error. Row-count differences remain explicit result data. Import verifies checksums before insertion. Existing rows are skipped according to the migration service’s idempotent import rules. The result contains per-table inserted and skipped counts plus warnings.

​
ClickHouse export

ClickHouse export requires the PostgreSQL sidecar manifest and a new destination directory:
The destination must not already exist. This lets the exporter remove the complete directory after failure without touching pre-existing files. The directory and streamed Parquet files use restrictive permissions and atomic temporary files. The export covers active session, checkpoint, layer, audit, security, and webhook tables. Older sources may omit tables. Telemetry is divided into deterministic time-and-hash chunks so each ClickHouse query and Parquet file stays bounded independently of the total dataset size. Oversized or memory-limited chunks are split again automatically. telemetry_manifest.json records every chunk’s range, shard, checksum, size, row count, and migration ID.

​
ClickHouse validation and import

Telemetry validation checks:
  • Parquet checksums
  • Manifest row counts against target ClickHouse when supplied
  • Agent and user references against target PostgreSQL when supplied
Checksum failure is fatal. Row-count differences and orphan groups are returned explicitly. Telemetry import is resumable per Parquet chunk. Successfully imported chunk IDs and checksums are recorded in .import_state.json in the input directory, so a retry skips only verified completed chunks rather than an entire table or month. Deterministic insert tokens protect ambiguous retries on current non-replicated MergeTree schemas. Imported project-keyed rows normalize to the deployment project default, and session summaries are rebuilt after event import so summaries reflect complete sessions rather than individual insert blocks.

​
Human and JSON behavior

All six leaves accept --output table|json. Human mode renders progress and summaries. JSON mode is finite, prompt-free, suppresses progress and warnings from stdout, and returns one result document. Failures leave stdout empty and emit one categorized error to stderr. Cleartext ClickHouse transport with credentials produces a human warning. JSON mode does not print a banner; operators should use clickhouses:// for TLS.

​
Exit codes