# Installation Observal has two parts: a **server** you self-host and a **CLI** installed on each developer machine. ## Install the server The server runs as a Docker Compose stack (API, web UI, PostgreSQL, ClickHouse, Redis, worker, load balancer). Prometheus and Grafana are optional deployment overlays. > \[!NOTE] Requires Docker Engine ≥ 24.0 with Compose v2 (`docker compose`, not `docker-compose`). Homebrew's Docker formula is outdated. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages. Verify with `docker version` and `docker compose version`. **One-line install:** ```bash curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install-server.sh | bash ``` This downloads a config package, runs guided setup (domain, secrets, ports), pulls container images from GHCR, and starts the full stack. **From source** (for contributors): ```bash git clone https://github.com/Observal/Observal.git && cd Observal cp .env.example .env make up ``` For deployment options, see [Self-Hosting](/self-hosting/docker-compose) and [Production deployment](/self-hosting/production-deploy). ## Install the CLI The CLI is what you use to log in, instrument harness configs, pull agents, and query traces. ## Install (standalone binary) The standalone binary is the simplest way to install. No Python required. ```bash curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash ``` This downloads the latest release binary for your platform and places it on your `PATH`. This validates the Ed25519-signed key, installs the CLI, and writes the key to `~/.observal/config.json`. If the key is invalid or expired, the installer exits with an error. Verify it worked: ```bash observal --version ``` ## Alternative: install with Python If you prefer to install via Python, use one of these methods. Requires Python 3.11 or newer. **uv (recommended):** ```bash uv tool install observal-cli ``` **pipx:** ```bash pipx install observal-cli ``` **pip:** ```bash pip install --user observal-cli ``` ### Optional extras Observal ships with two opt-in extras for the Python install: | Extra | What it adds | When to install | | --------- | ----------------------------------------- | ----------------------------------------------------------- | | `sandbox` | Docker SDK (for sandbox execution) | If you run agents inside Observal sandboxes | | `migrate` | `pyarrow` (for `observal server migrate`) | If you move registry and telemetry data between deployments | | `all` | Both of the above | If you do both | Install an extra: ```bash uv tool install 'observal-cli[sandbox]' ``` ## Install from source (for contributors) ```bash git clone https://github.com/Observal/Observal.git cd Observal uv tool install --editable . ``` ## What gets installed Four entry points land on your `PATH`: | Command | Purpose | | ---------------------- | -------------------------------------------- | | `observal` | The main CLI | | `observal-sandbox-run` | Sandbox runner invoked by Observal sandboxes | You will almost never call the sandbox runner directly. The CLI wires it into your harness config for you. ## Upgrade ```bash observal self upgrade ``` ## Uninstall Standalone binary: ```bash rm "$(which observal)" ``` Python install: ```bash uv tool uninstall observal-cli # or: pipx uninstall observal-cli # or: pip uninstall observal-cli ``` Uninstalling the CLI does **not** remove your config (`~/.observal/`). Delete that folder if you want a clean slate: ```bash rm -rf ~/.observal ``` ## Next -> [Quickstart](/getting-started/quickstart) # Quickstart Go from zero to "my first trace in the Observal dashboard" in about five minutes. This assumes you have Docker running. By the end of this guide you will have: * The Observal CLI installed * An Observal server running locally * The CLI logged in as an admin * At least one MCP server instrumented * A live trace visible in the web UI ## 1. Install the CLI ```bash curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash ``` No Python required. For alternative install methods, see [Installation](/). > \[!NOTE] You need Docker Engine ≥ 24.0 with Compose v2 (`docker compose`, not `docker-compose`). Homebrew's Docker formula is outdated. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages. Verify with `docker version` and `docker compose version`. ## 2. Start the server ```bash git clone https://github.com/Observal/Observal.git cd Observal cp .env.example .env docker compose -f docker/docker-compose.yml up --build -d ``` That's it. The `.env.example` ships with working defaults. The core services come up: | Service | URL | Purpose | | --------------------- | ----------------------- | ------------------------------ | | `observal-lb` (nginx) | `http://localhost` | Reverse proxy (API + Web) | | `observal-web` | `http://localhost:3000` | Web UI (Next.js, direct) | | `observal-api` | internal | FastAPI backend | | `observal-worker` | internal | Background jobs (arq) | | `observal-init` | internal | Runs DB migrations, then exits | | `observal-db` | `localhost:5432` | PostgreSQL 16 | | `observal-clickhouse` | `localhost:8123` | ClickHouse | | `observal-redis` | `localhost:6379` | Redis | Optional monitoring can be enabled with `make up-prometheus` or `make up-observability`. Prometheus listens on `http://localhost:9090`; Grafana listens on `http://localhost:3001` when the Grafana profile is enabled. The API waits for Postgres, ClickHouse, and Redis to pass health checks before starting. Expect 15–30 seconds. Confirm it is up: ```bash curl http://localhost/health # → {"status": "ok"} ``` Hitting a port conflict? See [Self-Hosting → Ports and volumes](/self-hosting/ports-and-volumes). ## 3. Log in ```bash observal auth login ``` Prompts: 1. **Server URL**: press Enter for `http://localhost` 2. **Login method**: pick `[E]mail` 3. **Email / password**: use one of the seeded demo accounts: | Role | Email | Password | | ----------- | ----------------------- | ------------------- | | Super Admin | `super@demo.example` | `super-changeme` | | Admin | `admin@demo.example` | `admin-changeme` | | Reviewer | `reviewer@demo.example` | `reviewer-changeme` | | User | `user@demo.example` | `user-changeme` | Log in as super admin for the fewest restrictions while exploring. Credentials land in `~/.observal/config.json` (mode `0600`). Check it worked: ```bash observal auth whoami # → super@demo.example (super_admin) ``` ## 4. Discover and instrument your harness If you already have MCP servers configured in Claude Code, Kiro, Cursor, VS Code, or Copilot, first see what's there: ```bash observal scan ``` Expected output lists detected harnesses, MCP servers, skills, hooks, and agents. MCP commands and remote URLs are shown exactly as configured. `scan` is read-only: it shows what you have without modifying anything. Install session telemetry hooks: ```bash observal doctor patch --all-harnesses ``` `doctor patch` installs supported session hooks and extensions. It does not rewrite MCP configuration. Restart your harness so the hook changes take effect, then begin a coding session. ## 5. See your first trace Open `http://localhost/traces` in your browser. Start a prompt in your harness and let the session complete. Refresh to see the indexed session and its parsed events. Or use the CLI: ```bash observal ops traces --limit 5 ``` ## 6. (Optional) Pull an agent Browse what the community has published: ```bash observal agent list observal agent show ``` Install one into your harness: ```bash observal agent pull --harness ``` This writes agent files, skills, hooks, and direct MCP configs into the right places for your harness. ## What you just built ```mermaid flowchart LR harness[Your harness] transcript[Local session transcript] hook[Session hook or extension] api[Session ingest API] ch[(ClickHouse)] ui[Web UI] harness --> transcript hook --> transcript hook --> api api --> ch ch --> ui ``` Observal indexes session transcript records into canonical events and aggregates. ## Where to next | You want to... | Go to | | ----------------------------------- | ---------------------------------------------------------------------- | | Understand registry identity | [Core concepts](/core-concepts/core-concepts) | | Understand session tracking | [Session tracking and reconciliation](/core-concepts/session-tracking) | | Learn what to do with traces | [Use Cases](/use-cases/use-cases) | | Configure the server for production | [Self-Hosting](/self-hosting/self-hosting) | | Deep-dive on a CLI command | [CLI Reference](/cli-reference/cli) | # Setup guide Connect your machine to Observal with two commands. It takes about two minutes and does not need Python or Docker. ## 1. Run the two commands for your OS **macOS / Linux** ```bash curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash observal auth login --server https://internal.observal.io ``` **Windows** (in PowerShell, not Command Prompt) ```powershell irm https://raw.githubusercontent.com/Observal/Observal/main/install.ps1 | iex observal auth login --server https://internal.observal.io ``` The first command installs the `observal` CLI. The second connects it to the server. ## 2. Answer the login prompts 1. **Login method**: pick the one your team uses. If unsure, choose **Web sign-in** and approve the code in your browser. 2. **`Fix all warnings? (configures telemetry and installs AI skills for detected harnesses) [Y/n]`**: answer **Y** (or press Enter). This is the step that instruments your coding tools. For each one Observal finds (Claude Code, Cursor, Kiro, Codex, Copilot, OpenCode, Pi, and others), it: * adds the hooks that send your sessions to Observal * installs the Observal skills, so your coding agent can search and use approved agents, MCP servers, and skills If you answer **n**, nothing is instrumented and your sessions won't show up. You can fix that later with `observal doctor patch --all-harnesses`. 3. Restart any coding tool that was open during setup so it picks up the changes. ## 3. Check it worked ```bash observal auth whoami ``` Then open [internal.observal.io](https://internal.observal.io). Your sessions appear under Traces after your next coding session. ## Troubleshooting | Problem | Fix | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `observal: command not found` | Open a new terminal window. On macOS/Linux, also check that `/usr/local/bin` is on your `PATH`. | | Intel Mac: download fails | There is no standalone binary for Intel Macs yet. Run `uv tool install observal-cli` instead of the first command. | | PowerShell refuses to run `irm ... \| iex` | Run `Set-ExecutionPolicy -Scope Process Bypass` in the same window, then retry. | | Login can't reach the server | Check that you're on the VPN, then run `curl https://internal.observal.io/health`. | | Answered **n** to the prompt, or added a new coding tool later | Run `observal doctor patch --all-harnesses`. | To upgrade later, run `observal self upgrade`. # Overview The vocabulary you need to be productive with Observal. ## The registry The registry stores users, agents, components, review state, and alert rules in PostgreSQL. Session events and aggregates are stored separately in ClickHouse. See [Session tracking and reconciliation](/core-concepts/session-tracking) for the complete session data flow. ## Registry components Six component types are available. Agents bundle the other five. | Type | What it is | | -------------- | ----------------------------------------------------------------------------------------------------------- | | **Agent** | A complete, installable AI agent. Bundles MCP servers, skills, hooks, prompts, and sandboxes into one YAML. | | **MCP Server** | A [Model Context Protocol](https://modelcontextprotocol.io/) server, the tools an agent can call. | | **Skill** | A portable instruction package agents load on demand. | | **Hook** | A lifecycle callback that runs on session start, tool use, session end, and other supported events. | | **Prompt** | A named, parameterized prompt template with variable substitution. | | **Sandbox** | A Docker execution environment for running code the agent generates. | Anyone can publish. Admin review controls what appears in the public listing, but your own items are usable immediately without approval. ## Canonical identity An agent or component has a canonical `namespace/slug` identity. The namespace is the owner's username. The slug is stable within that namespace. Display names can change without changing the canonical identity. UUIDs are the stable reconciliation key. The server supplies canonical namespace, slug, display name, qualified name, and review status. Installed versions in the local lockfile remain local pins. ## Versions and locks Agents and components are versioned. An approved version never changes. Every agent version pins each of its components to one exact version, recorded with a content digest. Installing that agent version always installs those component versions, even after newer ones are approved. Agent versions released before pinning are the exception: a component they never locked installs at its latest approved release, with a warning. Only the agent's author moves them, by releasing a new agent version, which goes through review like any other. Installs are pinned too. The first `observal agent pull` in a project records the agent version it installed in `observal.lock`, a file meant to be committed. Later pulls, by anyone in that project, install the same agent version until someone moves it with `--upgrade` or `--version`. `~/.observal/lockfile.json` is each machine's record of what it has installed where. `--strict` refuses any install that does not match its lock. See [`observal agent pull`](/cli-reference/pull#pinned-versions) and [`observal agent release`](/cli-reference/agent#release-and-versions). ## Deployment mode SSO-only access is controlled by `deployment.sso_only`: | Mode | Self-registration | Bootstrap | Auth | | ------------------------------------- | ----------------- | --------------------------------------------------- | ----------------------------- | | `deployment.sso_only=false` (default) | Yes | Yes, a fresh server creates an admin on first login | Email and password or API key | | `deployment.sso_only=true` | No | No | SSO only | Most self-hosters use `deployment.sso_only=false`. ## Next * [Session tracking and reconciliation](/core-concepts/session-tracking) * [Use cases](/use-cases/use-cases) # Session tracking and reconciliation Observal turns local harness conversations into indexed session events. This page explains the complete path from a harness source through durable delivery, server parsing, reconciliation, and recovery. ## Data model ### Session A session is one harness conversation or task identified by `session_id`. Observal records the harness, authenticated user, project, agent attribution, and source position with each record. ### Source record A source record is one complete item emitted by the harness. For a JSONL source, it is one non-empty, newline-terminated line. For a native source, the extension creates an equivalent stable record before delivery. ### Event The server parser converts each source record into an event such as a user prompt, assistant response, tool call, tool result, token update, lifecycle hook, error, or subagent event. Available detail depends on the harness source. ### Aggregate `session_stats_agg` summarizes event counts, prompts, tool calls, tool results, token totals, models, harnesses, agent attribution, and timing. Dashboards and insights query these aggregates. ## End-to-end flow ```mermaid flowchart TD source["Harness source"] wake["Hook, extension, or reconcile"] adapter["Harness adapter"] outbox["Durable local outbox"] ingest["Session ingest API"] records["Raw session records"] parser["Harness parser"] events["Parsed events and aggregates"] checkpoint["Contiguous server checkpoint"] source --> adapter wake --> adapter adapter --> outbox outbox --> ingest ingest --> records records --> parser parser --> events ingest --> checkpoint checkpoint --> outbox ``` The normal pipeline has seven stages: 1. The harness writes a JSONL transcript or exposes messages through a native API. 2. A hook, extension event, or `observal reconcile` wakes the exporter. 3. The harness adapter discovers the source and reads only complete records after the current checkpoint. 4. The exporter writes the batch to a durable local outbox before attempting the network request. 5. The exporter posts the batch to `POST /api/v1/ingest/session`. 6. The server stores raw records, parses them, and returns the highest contiguous acknowledged source position. 7. The exporter removes acknowledged batches and advances its local cursor. Hooks and reconciliation use the same source adapters and acknowledgement protocol. Reconciliation is a recovery path, not a separate ingestion system. ## JSONL sources JSON Lines stores one JSON value per line. Observal treats a line as ready only after its terminating newline is present. An incomplete final line remains local until the harness finishes writing it. Blank lines are ignored. The exporter retains three pieces of source position: * the zero-based record index * the absolute byte offset at the end of each record * the local byte cursor used for the next read Raw records are not normalized in the CLI. The server selects the parser registered for the harness, which keeps source discovery independent from event classification. OpenCode and Pi use native extension APIs rather than reading another process's transcript. Their extensions convert messages to stable JSONL-compatible records and then follow the same indexed ingest contract. ## Discovery and wake-ups Each harness adapter knows where that harness stores sessions and how to identify a stable session ID. Adapters may discover JSONL files, materialized hook records, or native message records. Hooks and extensions wake delivery at supported lifecycle boundaries, including prompt submission, idle, stop, and agent completion. A wake-up is only a request to resume work. Delivery state lives in the durable outbox and checkpoints, so correctness does not depend on one hook process staying alive. `observal agent pull` installs the appropriate session hooks or extension for the selected harness. `observal doctor` diagnoses missing instrumentation and can repair it with confirmation. ## Durable local delivery Session delivery is eventual at-least-once delivery with effectively-once storage. | Exporter | Durable pending data | Cursor or state | | ---------------------- | -------------------------------------- | --------------------------------------- | | Shared Python exporter | `~/.observal/telemetry_buffer.db` | `~/.observal/sync_state.json` | | OpenCode extension | `~/.observal/opencode_session_outbox/` | Per-session state in the same directory | | Pi extension | `~/.observal/pi_session_outbox/` | `~/.observal/sync_state.json` | The exporters cap their outboxes at 256 MiB. Reaching the limit raises or logs an explicit error instead of evicting unacknowledged records. A network timeout, process exit, or server outage leaves the pending batch intact. A later hook, extension event, or reconciliation run retries it. The shared exporter drains older pending records before sending newly discovered source records. Pending data is bound to its destination server and authenticated user. It is not silently delivered to a different registry or account after a login change. ## Checkpoints and idempotency The server identifies a source record by project, user, harness, session ID, and source index. Repeated or overlapping uploads therefore converge on the same record. An acknowledgement contains: * `acknowledged_line`, the highest contiguous source index stored by the server * `acknowledged_offset`, the corresponding source byte offset Only a contiguous acknowledgement advances the local cursor. If the server has indexes 0 and 2 but not 1, the checkpoint remains at 0. This prevents a later record from hiding a missing earlier record. If local cursor state is missing, invalid, or stale, the exporter reads the authenticated checkpoint from `GET /api/v1/ingest/session/checkpoint`. It resumes from the server position when that position matches the local source. This avoids replaying all acknowledged history after local state loss. If the content at an acknowledged source index changes, the server rejects the conflict rather than overwriting the canonical record. ## Finalization and integrity repair Normal hook uploads hash only the records needed for the current batch. They do not repeatedly hash the complete session. At finalization, the exporter makes a stable-file pass and sends the total record count, total byte offset, and SHA-256 session hash. The server compares that summary with stored records. If it finds a gap or mismatch, it returns `repair_from_line`. The exporter rewinds to that point and replays the affected range through the same idempotent path. This repair handles interrupted uploads and stale cursors. It cannot recover a source record that the harness deleted before any installed hook or extension observed and durably spooled it. ## Server processing The ingest endpoint validates batch size, line size, ordered byte offsets, finalization metadata, and authentication. It stores raw source records in ClickHouse `session_events` and classifies them with the registered harness parser. Session records include agent ID, agent version, and layer hash when attribution is available. Claude Code subagent records can also include a parent session ID. Kiro can report total session credits as durable metadata. After ingestion, session-level aggregates are available through `session_stats_agg`. PostgreSQL remains responsible for relational registry, user, review, and configuration data. MCP commands and remote URLs remain direct; Observal does not proxy MCP transport traffic. ## Reconciliation Run reconciliation when hooks were installed late, a machine was offline, delivery was interrupted, or you want to verify recent local history. Preview the default seven-day scan without sending data: ```bash observal reconcile --dry-run ``` Push recent sessions from every installed harness: ```bash observal reconcile ``` Limit discovery to one harness or time window: ```bash observal reconcile --harness kiro --since 24 ``` `--since` accepts 1 through 8,760 hours and defaults to 168. Without `--harness`, the command scans every installed adapter. A non-dry run validates its targets before touching the outbox, retries pending batches, recovers each server checkpoint, sends only records after that checkpoint, and finalizes fully uploaded but unfinished sessions. A dry run reports sessions whose local source is larger than the local cursor and sessions that still need final metadata. It does not drain the outbox, contact the ingest endpoint, or change cursor state. JSON results distinguish delivered, finalized, queued, rejected, skipped, and failed sources. ## Installation and verification Check authentication and local delivery health: ```bash observal auth status observal ops telemetry status ``` Preview recoverable sessions, push them, and inspect recent traces: ```bash observal reconcile --dry-run observal reconcile observal ops traces --limit 5 ``` If instrumentation is missing, run Doctor and confirm the proposed repair: ```bash observal doctor ``` ## Troubleshooting ### Sessions are not discovered Confirm that the harness is installed, its local session source exists, and the source was modified inside the `--since` window. Use `--harness` to isolate one adapter. ### Records remain pending Run `observal ops telemetry status`, verify the configured server is reachable, and confirm the current login matches the user and destination associated with the pending records. The next exporter wake-up or `observal reconcile` retries them. ### A checkpoint does not match the local source The command leaves the source untouched and reports the mismatch. Preserve both the source and `~/.observal/` state while investigating. A changed or truncated transcript can no longer prove the byte position previously acknowledged by the server. ### A record is permanently rejected The shared exporter quarantines permanently rejected payloads in `~/.observal/telemetry_buffer.rejected.jsonl` so one invalid session does not block later sessions. Inspect the rejection reason and source record before retrying or removing it. ### Hooks are missing Run `observal doctor`. Doctor reports the exact metadata or harness-file changes and asks for confirmation before applying repairs. # Overview Observal solves five concrete problems in the AI-agent lifecycle. Each page below is written as a playbook: what the problem is, how Observal addresses it, and the exact commands to run. | If you need to... | Read | | ------------------------------------------------ | ---------------------------------------------------------------------- | | See what your MCP servers are actually doing | [Observe MCP traffic](/use-cases/observe-mcp-traffic) | | Package an agent once and ship it to any harness | [Share agent configs across harnesses](/use-cases/share-agent-configs) | | Figure out why a session went wrong | [Debug agent failures](/use-cases/debug-agent-failures) | | Give your whole team a source of truth | [Run a team-wide agent registry](/use-cases/team-registry) | ## How these relate ```mermaid flowchart LR observe[Observe MCP traffic] debug[Debug agent failures] share[Share agent configs] registry[Team registry] observe -->|traces give you evidence| debug observe --> share share -->|governance + reuse| registry ``` You typically adopt Observal in that order: **observe** first (low-risk, instant value), then **debug**, then **share**, and finally **run a team registry** once you have enough published agents to justify governance. # Observe MCP traffic Observal identifies MCP and tool activity from the session transcript recorded by each coding harness. It does not intercept MCP transport traffic. ## What you get When the harness records the relevant detail, session events can include: * Tool or MCP name * Tool input and result * Event ordering within the session * Harness, user, agent, and model attribution * Token totals and timing derived from the session Coverage varies by harness because Observal can only index fields present in the local transcript or native message API. Transport-level latency and payload data are not synthesized when the harness does not record them. ## Discover and install session hooks ```bash observal scan observal doctor patch --all-harnesses ``` `scan` is read-only. `doctor patch` installs supported session hooks or extensions and leaves every MCP command and remote URL unchanged. Scope hook installation to selected harnesses: ```bash observal doctor patch --harness claude-code observal doctor patch --harness kiro observal doctor patch --harness copilot-cli ``` Restart the harness after hook changes, then run a coding session. If a hook was missed or the machine was offline, reconcile local session sources: ```bash observal reconcile ``` ## Query collected sessions Open `/traces` in the web UI to filter sessions by harness, agent, user, model, and time range. Expand a session to inspect parsed prompts, responses, tool calls, tool results, and lifecycle events. The CLI can list recent sessions and unfold their events: ```bash observal ops traces --limit 20 observal ops traces --turn --limit 10 ``` ## Caveats * MCP visibility depends on the harness transcript format. * Observal does not modify, proxy, or wrap MCP traffic. * Monetary cost and structured transport errors are unavailable unless the harness records equivalent session fields. ## Next [Debug agent failures](/use-cases/debug-agent-failures) explains how to investigate failures using session evidence. # Share agent configs across harnesses Your team has a reviewer agent that works great in Claude Code. Now someone wants to use it in Kiro. Then Cursor. Then they want to tweak the skills. Copy-pasting config snippets across tools doesn't scale. Observal's registry gives you one agent definition that installs cleanly into every supported harness. ## The shape of an agent Every agent is a YAML file that bundles: * MCP servers it needs * Skills to load * Hooks to wire into the session lifecycle * Prompts (with variables) * Sandboxes for code execution When someone runs `observal agent pull `, Observal templates that YAML into the right files for their harness: `~/.claude/agents/*.json`, `.kiro/agents/*.json`, `.cursor/mcp.json`, and so on. ## Publish an agent ### Option A - the interactive wizard ```bash observal agent create ``` Step-by-step prompts: name, description, which MCP servers, which skills, which hooks. Results in a registry entry you can share by ID. ### Option B - the YAML workflow (recommended for teams) ```bash observal agent init # scaffold observal-agent.yaml observal agent add mcp github-mcp # add components observal agent add skill code-review-skill observal agent add hook pretooluse-logger observal agent build # validate (dry-run) observal agent publish # submit to registry ``` The YAML workflow is PR-reviewable. The file lives in your repo; changes flow through your normal review process. ## Install an agent into any harness Browse what exists: ```bash observal agent list observal agent list --search review observal agent show ``` Install with one command, pick the harness: ```bash observal agent pull --harness claude-code observal agent pull --harness kiro observal agent pull --harness cursor observal agent pull --harness vscode observal agent pull --harness codex ``` The CLI prompts for any environment variables the MCP servers declare as required (GitHub tokens, API keys). These are stored in your harness config, not uploaded to Observal. ### Control what gets installed ```bash # Preview without writing anything observal agent pull --harness claude-code --dry-run # Install into a specific directory observal agent pull --harness claude-code --dir ./my-project # Claude Code only: scope (project-local vs user-global) observal agent pull --harness claude-code --scope project observal agent pull --harness claude-code --scope user # Claude Code only: sub-agent model observal agent pull --harness claude-code --model sonnet # Claude Code only: tool allowlist observal agent pull --harness claude-code --tools Read,Write,Bash ``` ## What portability actually means The harness feature matrix (defined in `packages/observal-shared/observal_shared/harness_registry.py`) controls what each harness supports. If an agent uses skills and the target harness doesn't have skills, the installer: * Installs the compatible parts cleanly * Warns about the unsupported parts * Exits non-zero if the agent *requires* something the harness cannot provide Useful when onboarding a new machine or swapping between "work setup" and "personal setup." ## Next → [Run a team-wide agent registry](/use-cases/team-registry): once publishing is routine, you need governance. # Debug agent failures AI failures rarely produce one reliable error code. Observal preserves the session transcript so you can inspect what the user asked, which tools ran, what results the harness recorded, and how the agent responded. ## 1. Find the session Open `/traces` in the web UI or list recent sessions: ```bash observal ops traces --limit 50 ``` Filter by harness, agent, user, model, or time range. Select the session that matches the reported failure. ## 2. Inspect the event sequence Expand the session to review prompts, assistant responses, tool calls, tool results, lifecycle events, and subagent activity in source order. The CLI can unfold the same session structure: ```bash observal ops traces --turn --limit 10 ``` Look for repeated tool calls, error results, missing inputs, unexpected model responses, or a stop event that occurred before the expected work completed. ## 3. Inspect recorded tool detail The detail available for a tool call depends on the harness transcript. When present, Observal displays the tool name, input, result, and error indicators. Observal does not intercept MCP transport traffic or invent payload fields that the harness did not record. ## 4. Common patterns | Session pattern | Likely cause | | -------------------------------------------------------- | ------------------------------------------------------------------------------ | | Same tool called repeatedly with identical arguments | The agent is stuck in a retry loop or is not consuming the result | | Tool result records an error and the assistant continues | Prompt or recovery policy did not handle the failure | | Tool call has no matching result | Harness shutdown, interrupted process, or incomplete transcript delivery | | Session stops immediately after a prompt | Hook, model, permission, or harness configuration issue | | Expected MCP call never appears | The model did not select the tool, or the harness transcript omits that detail | ## 5. Recover incomplete delivery If the local harness still contains the session source, run: ```bash observal reconcile ``` Reconciliation resumes from the acknowledged checkpoint and replays missing source records idempotently. # Run a team-wide agent registry Once two or more people are authoring agents, you need a single source of truth. Observal becomes your team's internal Docker Hub for AI agents, with review, RBAC, and telemetry baked in. ## What changes at team scale * **Discovery**: everyone sees the same list of agents, MCPs, skills, hooks, prompts, and sandboxes. * **Review**: admins approve what appears in the public listing. Authors' own items are still immediately usable. * **Governance**: RBAC roles (`super_admin`, `admin`, `reviewer`, `user`) control who can publish and approve. * **Visibility**: centralized dashboards instead of "ask Sarah which version she's running." ## Setup shape Deploy once, everyone points at it. ```mermaid flowchart TB server["Observal server - API + Web UI"] db[(PostgreSQL)] ch[(ClickHouse)] redis[(Redis)] a["Engineer A - Claude Code"] b["Engineer B - Kiro"] c["Engineer C - Cursor"] server --> db server --> ch server --> redis a --> server b --> server c --> server ``` Install the server once ([Self-Hosting](/self-hosting/self-hosting)). Then every engineer installs the CLI and runs `observal auth login` pointed at your shared server URL. ## Users and roles Four roles, RBAC-enforced on every endpoint. | Role | Can | Cannot | | ------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `user` | Publish components (subject to review), install agents, view their own traces | Approve submissions, see other users' private traces, change server settings | | `reviewer` | Everything `user` can + approve/reject submissions | Change server settings, manage users | | `admin` | Everything `reviewer` can + manage users, change server settings | Only restriction: certain super-admin operations | | `super_admin` | Everything | - | Manage users: ```bash observal admin users --output json observal admin reset-password --generate --output json observal admin delete-user --force --output json ``` Change a role via the web UI (`/settings/users`) or the API (`PUT /api/v1/admin/users/{id}/role`). ## Onboarding a new engineer Two commands to get them productive: ```bash # The new engineer runs: curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash observal auth login --server https://observal.your-company.internal ``` For managed deployments, users authenticate through SSO or are provisioned by an admin. See [Authentication and SSO](/self-hosting/authentication). After logging in, they can: ```bash observal agent list # see every agent the team has published observal agent pull team-reviewer --harness claude-code # install one observal scan # discover what they have installed observal doctor patch --all-harnesses # instrument everything ``` ## Review workflow Authors submit. Reviewers approve. Approved items appear in the public listing. ```bash observal admin review list --output json observal admin review show --output json observal admin review approve --output json observal admin review reject --reason "missing env var docs" --output json ``` What reviewers look for: * Does the README/description make it clear what the component does? * Does the MCP analysis (from `submit`) look correct: tools, env vars, transport? * Are required env vars documented? * Is the repo URL trustworthy (pinned commit or tag)? Everything published is visible to the author immediately. Review controls what appears in the public listing. ## Telemetry across the whole team Because every engineer's session telemetry flows into the same server, `observal ops` becomes a team dashboard: ```bash observal ops top --type agent # most-used agents across the team observal ops top --type mcp # hottest MCP servers ``` Filters in the web UI let you slice by user, agent, harness, and time range. ## SSO and audit considerations For orgs that need SSO and audit logging, configure OIDC in **Admin → SSO**: | Setting | Value | | --------------------------- | ---------------------- | | `oauth.client_id` | Your IdP client ID | | `oauth.client_secret` | Your IdP client secret | | `oauth.server_metadata_url` | Your IdP discovery URL | Restart the API after OIDC changes. See [Authentication and SSO](/self-hosting/authentication). ## Next → [Self-Hosting](/self-hosting/self-hosting): the operator's playbook for actually running the server this use case depends on. # Teamspaces A teamspace is a shared registry namespace with a unique handle such as `platform-tools`. Team handles and usernames cannot collide. ## Roles Every member has one role: | Role | Can | | ---------- | --------------------------------------------------------------- | | `owner` | Manage members, invites, visibility, and registry content | | `reviewer` | Approve or reject the teamspace's pending agents and components | | `member` | View the roster and publish into the teamspace | Global admins can manage any teamspace. A teamspace always keeps at least one owner. ## Creating a teamspace Any signed-in user can create a teamspace under **Registry, Teamspaces**. Private teamspaces are ready immediately. A public teamspace reserves its handle but stays private and locked until a global reviewer or above approves it in the Review queue. A global reviewer may approve their own request. Changing an existing private teamspace to public follows the same review flow. A rejected request keeps the teamspace private and can be submitted again. Personal teamspaces always remain private. ## Publishing and review Private teamspaces can publish only team-private agents and components. Teamspace submissions never auto-approve. After a teamspace is approved for public visibility, its owners and team reviewers may approve or reject all content in that namespace, including public content and their own submissions. Global reviewers continue to review public content across the registry. Public submissions notify both groups, while team-private submissions notify only that teamspace's owners and reviewers. Global admins may still review all content. A public teamspace cannot become private while it owns public agents, components, or component sources. Restrict or remove those items first. ## Membership Owners and global admins can add users and assign roles. Public teamspaces accept join requests. Private teamspaces use expiring invite links, and their owners decide the resulting requests. Members can leave unless they are the last owner. # Overview Complete reference for the `observal` CLI. Every subcommand has its own page; this overview is the index. > **New to Observal?** Start with [Quickstart](/getting-started/quickstart) and come back here when you need a specific command. ## Command groups | Command | What it does | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | [`observal api`](/cli-reference/api) | Call authenticated JSON endpoints without a dedicated command | | [`observal auth`](/cli-reference/auth) | Authentication and account management | | [`observal config`](/cli-reference/config) | Local CLI configuration, aliases | | [`observal discover`](/cli-reference/discover) | Find approved resources for a task and use them in the current session | | [`observal scan`](/cli-reference/scan) | Discover what's installed across your harnesses (read-only) | | [`observal share`](/cli-reference/share) | Share and pull scoped, version-pinned repository Agent sets | | [`observal outdated`](/cli-reference/outdated) | Compare installed agent and component versions with the active registry | | [`observal reconcile`](/cli-reference/reconcile) | Backfill sessions missed by automatic telemetry delivery | | [`observal inbox`](/cli-reference/inbox) | Read and update the signed-in user's work and event feed | | [`observal agent pull`](/cli-reference/pull) | Install a published agent into a harness | | [`observal registry`](/cli-reference/registry) | Publish and manage components (MCP / skill / hook / prompt / sandbox) | | [`observal registry recommend`](/cli-reference/recommend) | Get personalized component recommendations | | [`observal registry version`](/cli-reference/component) | Publish and inspect component versions | | [`observal registry models`](/cli-reference/models) | Inspect packaged harness model catalogs | | [`observal agent`](/cli-reference/agent) | Create, install, and manage agents | | [`observal team`](/cli-reference/team) | Manage teamspaces, members, join requests, and invitations | | [`observal ops`](/cli-reference/ops) | Observability and operations (sessions, telemetry, logs, insights) | | [`observal admin`](/cli-reference/admin) | Core administration and submission review | | [`observal doctor support`](/cli-reference/doctor/support) | Generate and inspect redacted diagnostic bundles | | [`observal doctor`](/cli-reference/doctor) | Diagnose harness compatibility; `doctor patch` applies instrumentation | | [`observal server migrate`](https://github.com/Observal/Observal/blob/main/docs/cli/migrate.md) | Export/import PostgreSQL registry (shallow copy) and ClickHouse telemetry (deep copy) | | [`observal self`](/cli-reference/self) | Upgrade or downgrade the CLI | | [`observal registry prompt`](/cli-reference/prompt) | Manage reusable prompts in the registry | | [`observal server`](/cli-reference/server) | Manage the embedded server (start, stop, upgrade, rollback) | | [`observal registry skill`](/cli-reference/skill) | Submit, browse, and install portable skill packages | ## Global options Any subcommand accepts these. | Option | Short | Description | | ----------- | ----- | --------------------------------------- | | `--version` | `-V` | Print the CLI version and exit | | `--verbose` | `-v` | Verbose output | | `--debug` | - | Debug-level logging (extremely verbose) | | `--help` | - | Show help for any command or subcommand | ## JSON list contract Every dedicated list command returns the same envelope: ```json { "items": [], "total": 0, "page": 1, "page_size": 0 } ``` Paginated commands preserve server totals and requested pages. Unpaginated commands use page 1 and the returned item count as `page_size`. Detail and mutation commands return direct objects. `observal api` intentionally preserves raw endpoint JSON. ## Exit codes Consistent across all commands: | Code | Meaning | | ---- | -------------------------------------------- | | 0 | Success | | 1 | Unexpected or uncategorized failure | | 2 | Usage error | | 3 | Authentication required or failed | | 4 | Permission denied | | 5 | Resource not found | | 6 | Conflict with current state | | 7 | Validation failure | | 8 | Rate limit reached | | 9 | Network, service, or dependency unavailable | | 10 | CLI and server version mismatch | | 11 | Batch completed with one or more item errors | Errors identify the failed operation, resource, remediation, and server request ID when available. Repairable failures may include a safe structured `result`; internal details appear only with `--debug`. When JSON output is selected, errors are written to stderr as one JSON object and stdout remains clean. Startup migrations and update notices never write to JSON stdout. A nonfatal startup migration warning is a structured `warning` object on stderr. ```json { "error": { "category": "not_found", "message": "The requested resource was not found.", "operation": "Show agent", "resource": "agent registry", "remediation": "Check the identifier or list available agents.", "request_id": "01J...", "http_status": 404, "exit_code": 5 } } ``` ## Non-interactive mode For scripts and CI, pair flags with environment variables: ```bash export OBSERVAL_SERVER_URL=https://observal.your-company.internal export OBSERVAL_API_KEY= observal ops traces --limit 100 --output json | jq ``` Full env var reference: [Environment variables](/reference/environment-variables). ## Output formats Read-heavy commands (`list`, `show`, `traces`, `spans`) support `--output`: ```bash observal registry mcp list --output table # default, TTY-friendly observal registry mcp list --output json # machine-readable ``` ## Aliases IDs get long fast. Create shortcuts: ```bash observal config alias my-mcp 498c17ac-1234-4567-89ab-cdef01234567 observal registry mcp show @my-mcp ``` See [`observal config`](/cli-reference/config) for details. ## Next → [`observal auth`](/cli-reference/auth): you'll need to log in first. # observal api Call an authenticated JSON endpoint when no dedicated high-level command exists. ## Examples ```bash observal api GET /api/v1/teams --output json observal api GET /api/v1/agents --param tag=review --param tag=security --output json observal api POST /api/v1/teams --from-file team.json --output json cat team.json | observal api POST /api/v1/teams --output json ``` Methods are `GET`, `POST`, `PUT`, `PATCH`, and `DELETE`. Paths must be canonical relative `/api/v1/` paths. Full URLs, encoded or plain traversal segments, fragments, and inline query strings are rejected. Use repeatable `--param KEY=VALUE` options for query parameters; repeated keys preserve their original order. The command uses the configured bearer token. It does not accept arbitrary authorization headers, so it cannot forward credentials to another host. ## Request bodies `POST`, `PUT`, `PATCH`, and `DELETE` accept any single JSON value from `--from-file` or standard input: objects, arrays, strings, numbers, booleans, and explicit `null`. A file takes precedence when both are present. `GET` rejects every request body, including `null`. ## Output Table mode renders arbitrary objects as field and value rows. JSON mode preserves the endpoint response exactly, including top-level arrays. This raw behavior is intentional and is the exception to dedicated list commands, which use the standard list envelope. ## Errors and retries The command uses the shared categorized error contract and preserves server request IDs. Automatic transient retries apply only to `GET`. After an uncertain mutation failure, read endpoint state before retrying. See [Mutation retries and idempotency](/cli-reference/idempotency). Prefer dedicated commands when they exist because they provide stronger validation, safer confirmation, and domain-specific output. # Mutation retries and idempotency Agents must distinguish a rejected mutation from a mutation whose result is unknown. ## Automatic retry contract The shared CLI client automatically retries authenticated `GET` requests on HTTP 429, 503, and 504 responses, honoring `Retry-After` when present. It attempts token refresh once after a 401 response. `POST`, `PUT`, `PATCH`, and `DELETE` requests are sent once. The CLI does not retry transient mutation responses because the server may have applied the change before the connection failed. ## Agent workflow 1. Read current state and capture UUIDs, canonical names, versions, or status. 2. Send one mutation. 3. If the command succeeds, inspect the returned state and perform the smallest verification read when needed. 4. If validation, permission, or conflict is returned, correct that state rather than retrying unchanged. 5. If timeout, connection failure, 503, or 504 occurs after a mutation was sent, treat the result as unknown. Read current state before deciding whether to retry. ## Mutation families | Mutation | Verification before retry | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Create Agent or component | List or show by canonical `namespace/slug`; do not create again if it exists | | Publish or release version | Read version history and review status | | Update visibility, role, or settings | Read the specific resource and compare the desired field | | Approve or reject a request | Read request status by UUID | | Archive, restore, revoke, or delete | Show the resource or confirm not-found where deletion is expected | | Install or pull | Inspect returned files and run `scan` for the target harness | | Bulk submit | Re-run only after reviewing per-entry results; existing identities return conflicts and are reported as skipped | | Server upgrade, rollback, or migration | Read service version, migration validation, and destination state before any retry | ## Conflict behavior A conflict is deterministic information, not a transient failure. Common decisions include: * Use `qualified_name` when a bare name is ambiguous. * Use update only when the user requested an in-place change. * Use release or version publish when the user requested a new version. * Treat an already-created bulk component as skipped after verifying its identity. * Do not bypass edit locks or ownership conflicts. The `observal api` escape hatch follows the same retry rules because it uses the shared authenticated client. # observal auth Authentication and account management. ## Commands | Command | Description | | ------------------------------- | ------------------------------------------------------------------- | | `observal auth login` | Authenticate with credentials or browser SSO | | `observal auth logout` | Revoke the remote session when possible and clear local credentials | | `observal auth whoami` | Show the authenticated user | | `observal auth status` | Check authenticated server and local outbox health | | `observal auth change-password` | Change the current user's password | | `observal auth set-username` | Set or update the registry namespace username | ## Login ```bash observal auth login observal auth login --server https://observal.example.com --email alice observal auth login --sso observal auth login --sso --provider google ``` Login accepts `server`, `email`, `password`, `name`, `sso`, `saml`, `provider`, `output`, and `no-setup` options. Prefer `OBSERVAL_PASSWORD` or `OBSERVAL_PASSWORD_FILE` over the password option so the secret does not enter shell history or process arguments. Human login always asks for the server URL unless `--server` is supplied; leave the prompt blank to use `http://localhost`. On a fresh server, provide email, name, and a password to create the first administrator. JSON mode never prompts, uses the configured server or local default, and requires complete credential inputs. Successful human login synchronizes the bundled skills, creates the initial layer snapshot, and runs doctor. Select `no-setup` to skip the snapshot and doctor. JSON mode skips those post-login steps. `provider` picks the browser SSO provider: `oidc`, `saml`, `google` or `github`. It implies `sso`, and the CLI rejects a provider the server has not enabled, or `saml` combined with a different `provider`. A bare `sso` uses the only enabled provider, or asks which one when the server enables several. JSON mode and non-interactive input never ask, so a bare `sso` there falls back to the server default (OIDC, then SAML); pass `provider` to choose. Every CLI invocation also computes a SHA-256 hash for each installed Observal-managed skill tree. A mismatched tree is replaced completely from the packaged bundle, including references and scripts, so local edits and stale extra files do not survive. Skill directories outside the six bundled Observal names are untouched. Credential JSON login emits one safe object and never includes tokens or passwords: ```bash OBSERVAL_PASSWORD_FILE=/run/secrets/observal-password \ observal auth login \ --server https://observal.example.com \ --email alice \ --output json ``` Browser SSO in JSON mode is a JSON Lines stream. The first event contains the verification URL and user code. The final event confirms authentication: ```bash observal auth login --sso --output json ``` ## Logout ```bash observal auth logout observal auth logout --output json ``` Logout always removes local tokens when the local configuration is readable. Remote revocation is best effort and is reported separately in JSON. Use `observal doctor cleanup` to remove Observal-managed harness hooks. ## Current user ```bash observal auth whoami observal auth whoami --output json ``` The JSON form returns the server user object directly. ## Status ```bash observal auth status observal auth status --output json ``` Status reports the server URL, authentication state, health latency, and local telemetry outbox. It returns exit code 3 when authentication is absent and exit code 9 when the configured server is unreachable. ## Change password ```bash observal auth change-password ``` Both modes read `OBSERVAL_CURRENT_PASSWORD` and `OBSERVAL_NEW_PASSWORD`, including their corresponding `_FILE` forms. Human mode prompts for missing values; JSON mode requires both values and never prompts. ```bash OBSERVAL_CURRENT_PASSWORD_FILE=/run/secrets/current-password \ OBSERVAL_NEW_PASSWORD_FILE=/run/secrets/new-password \ observal auth change-password --output json ``` Passwords require at least 12 characters, one uppercase letter, one number, and one special character. ## Set username ```bash observal auth set-username alice observal auth set-username alice --output json ``` The username is the user's registry namespace. It must follow the namespace rules shown by command help. The server prevents changes that conflict with another namespace or published registry ownership. ## Environment variables | Variable | Purpose | | --------------------------- | ------------------------------------------------------------------ | | `OBSERVAL_SERVER_URL` | Default server URL | | `OBSERVAL_ACCESS_TOKEN` | Pre-authenticate commands without login | | `OBSERVAL_PASSWORD` | Credential or bootstrap login password | | `OBSERVAL_CURRENT_PASSWORD` | Current password for JSON password changes | | `OBSERVAL_NEW_PASSWORD` | New password for JSON password changes and mandatory login changes | | `OBSERVAL_TIMEOUT` | Authenticated client timeout | Password and token variables support a corresponding `_FILE` form. Do not set a direct value and its file form together. Full list: [Environment variables](/reference/environment-variables). ## Related * [`observal config`](/cli-reference/config): local configuration * [Self-hosting authentication](/self-hosting/authentication): server authentication and SSO # observal config Manage local CLI configuration. Configuration lives in `~/.observal/config.json`; aliases live in `~/.observal/aliases.json`. Both files are written atomically with mode `0600`. ## Commands | Command | Description | | ------------------------- | --------------------------------------------------------- | | `observal config show` | Show effective configuration without exposing credentials | | `observal config set` | Set a validated user-managed value | | `observal config path` | Print the configuration file path | | `observal config alias` | Set or remove a local reference alias | | `observal config aliases` | List local aliases | Every command supports explicit table and JSON output. Human-readable table output is the default. `config path` preserves a bare path in table mode for shell composition. ## Show configuration ```bash observal config show observal config show --output json ``` Token values and fragments are never displayed. Output reports only whether access, refresh, or harness-hook credentials are configured. Environment overrides are included because the command shows effective configuration. ## Set configuration ```bash observal config set server_url https://observal.example.com observal config set timeout 60 --output json observal config set update_check false observal config set update_check_interval 86400 observal config set update_check_repo Observal/Observal ``` Only these user-managed keys are accepted: | Key | Accepted value | | ----------------------- | ------------------------------------------------------ | | `server_url` | HTTP or HTTPS URL without embedded credentials | | `timeout` | Positive integer seconds | | `update_check` | `true`, `false`, `yes`, `no`, `on`, `off`, `1`, or `0` | | `update_check_interval` | Integer seconds, minimum 60 | | `update_check_repo` | `owner/repository` or an empty string | Authentication and identity fields are managed by `observal auth` and cannot be set here. The removed `output` and `color` settings had no runtime consumers. Select JSON explicitly for each command. JSON output distinguishes the persisted value from the effective value, which may be overridden by an environment variable. ## Configuration path ```bash observal config path observal config path --output json ``` Table mode prints only the absolute path. JSON includes the path and whether the file currently exists. ## Set or remove an alias ```bash observal config alias reviewer alice/reviewer observal config alias reviewer alice/reviewer --output json observal config alias reviewer ``` Alias names must start with a letter and may contain letters, numbers, dots, underscores, and hyphens, up to 64 characters. Targets may be UUIDs, canonical `namespace/slug` identities, names, or other references accepted by the destination command. Omitting the target removes the alias. Removing an already absent alias is an idempotent success with `changed` set to `false` in JSON. ## List aliases ```bash observal config aliases observal config aliases --output json ``` JSON returns an `items` array and `total`. Empty output preserves the same shape. ## Environment overrides Environment values take precedence over persisted values for the current invocation. `config set` changes the file but does not replace an active environment override. See [Environment variables](/reference/environment-variables) for supported overrides. ## Related * [Config files](/reference/config-files): local file schemas and permissions * [`observal auth`](/cli-reference/auth): authentication-managed values # observal discover Find approved registry resources for the task at hand and use them in the current session, without building an Agent first. Discovery searches every kind at once (agents, MCP servers, skills, hooks, prompts, sandboxes), ranks by relevance, and tells you separately whether each result is approved, whether it works in your harness, and whether it can be used right now or only after a restart. For signed-in users, results are scoped to personal listings they own and listings in every teamspace they currently belong to (including public teamspace listings); other people's public listings are not returned. Global reviewers and admins retain their review/admin scope. Anonymous public discovery is unchanged when the public registry is enabled. The bundled `observal` skill tells coding assistants to run this before reinventing something or declaring a capability unavailable. ## Search ```bash observal discover search review a pull request for authentication bugs observal discover search 'query postgres' --type mcp --output json observal discover search generate tests --harness pi observal discover search release notes --include-unapproved ``` Pass the task as one shell-escaped argument (single-quote it and escape any embedded `'` as `'\''`). Several bare words are still joined into one query, but the text comes from the user and must never be spliced into a shell command unquoted. | Option | Description | | ---------------------- | ------------------------------------------------------------------- | | `--type`, `-t` | Restrict to `agent`, `mcp`, `skill`, `hook`, `prompt`, or `sandbox` | | `--harness`, `-i` | Only resources that list this harness as supported | | `--limit`, `-n` | Number of results, 1 through 25 (default 5) | | `--include-unapproved` | Also return your own pending or rejected submissions | | `--output`, `-o` | Table or JSON output | JSON returns `query`, `harness`, `count`, and `results[]`. Important result fields: | Field | Meaning | | ------------------------ | ----------------------------------------------------------------------------------------------- | | `identifier` | Permanent `urn:air:::` used by `inspect` and `use` | | `score` | Relevance to the query, 0 to 100. Never approval or trust. | | `matchedOn` | The query words that matched | | `obs:kind` | `agent`, `mcp`, `skill`, `hook`, `prompt`, `sandbox` | | `obs:nativeRef` | `namespace/slug@version`, usable with the registry and agent commands | | `obs:approval` | `approved`, `pending`, `rejected`, `archived`, `draft` | | `obs:availability` | `now`, `next-session`, `explicit-install`, `not-approved`, `archived`, `unsupported-in-harness` | | `obs:supportedHarnesses` | Harnesses the publisher declared; empty means unrestricted | | `obs:artifactDigest` | SHA-256 of the exact artifact bytes | An empty `results` array is a successful answer: nothing matched. Try fewer or different words before concluding the resource does not exist. ## Inspect ```bash observal discover inspect urn:air:observal.acme.com:skill:5f2c... --output json ``` Returns the complete entry, including `representativeQueries` (what the resource is good for), `capabilities`, the versioned artifact `url`, and its digest. The predecessor `urn:ai:` prefix is accepted. ## Use ```bash observal discover use urn:air:observal.acme.com:skill:5f2c... observal discover use urn:air:observal.acme.com:mcp:9a1b... --harness kiro --output json ``` | Option | Description | | ----------------- | ---------------------------------------------------------------------------------------------------------- | | `--harness`, `-i` | Harness this session runs in. Detected from `OBSERVAL_HARNESS` or the only installed harness when omitted. | | `--max-chars` | Cap on printed content (default 32000); longer artifacts are truncated with a pointer to the full URL | | `--yes`, `-y` | Load a resource that is not approved (your own drafts) | | `--output`, `-o` | Table or JSON output | What happens depends on the kind: * **Skills and prompts** are text. The exact approved version is fetched from its permanent URL and printed (JSON: `content`, `activated: true`, `mode: context`). The use is recorded in the capability lock. * **MCP servers, agents, sandboxes, and hooks** change your setup. Nothing is written; the response carries `next_step`, the existing install command (`observal registry mcp install …`, `observal agent pull …`), which confirms before touching harness config and takes effect after a restart. JSON: `activated: false`, `mode: next-session`. Unapproved resources are refused without `--yes`. A resource whose `obs:supportedHarnesses` excludes the detected harness is refused; pass `--harness` to override the detection. ## The capability lock Every activation appends one line to `~/.observal/capability_lock.jsonl`: timestamp, harness, working directory, identifier, kind, version, digest, and mode. `agent pull` and the component install commands append too. Session upload (`session_push` hooks and `observal reconcile`) attaches the lines that fall inside a session's harness, directory, and time window as `capabilities_used`, which is how Observal knows which resources a session relied on. Nothing secret is stored. Entries older than 30 days are pruned. Related: [`observal agent init --from-capabilities`](/cli-reference/agent) turns the resources used in a directory into a draft Agent. ## Exit codes Follows the shared CLI error contract: `4` permission (unapproved resource without `--yes`), `5` not found, `7` validation (unknown type, harness, or identifier). # observal scan Discover MCP servers, hooks, and telemetry configuration across your harness configs. `scan` is **read-only** -- it shows what you have without modifying any files. To install session telemetry hooks, use [`observal doctor patch`](/cli-reference/doctor). MCP commands and URLs are never rewritten. ## Synopsis ```bash observal scan [--harness ] [--inventory] [--output table|json] ``` Use `--inventory` for **local-only**, bounded harness evidence. It returns a versioned JSON document with `inventory` and `diagnostics`, or a table showing the harness, scope, component type, name, and privacy-safe source path. URLs have userinfo, fragments, and **all query parameters** removed; suspicious credentials in the URL path make a launch incomplete. Token values, raw prompts, and command arguments are not printed. Unrecognized or unsafe launches have `launch: null` and a diagnostic, not a guessed Registry identity. An empty inventory succeeds with an empty list. It never contacts Observal, writes a lockfile, or submits a component. Package-manager installations alone are not scanned or treated as configured capabilities. Component names and source filenames are user-supplied, so review the output before sharing it externally. To find approved resources for a task, use [`observal discover`](/cli-reference/discover). To submit something you found locally, **you** choose to run the corresponding `observal registry submit --draft` command; inventory never starts that workflow. ## Options | Option | Description | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `--harness ` | Scope to one harness: `cursor`, `kiro`, `claude-code`, `codex`, `copilot`, `copilot-cli`, `opencode`, `antigravity`, `goose`, `pi` | | `--inventory` | Bounded, redacted **local-only** evidence; no Registry lookup. | | `--output table\|json` | Render a table or machine-readable JSON. | If you run `observal scan` with no flags, it auto-detects every installed harness and scans each in turn. ## What it does 1. Finds MCP config files: * Claude Code: `~/.claude/settings.json` * Kiro: `.kiro/settings/mcp.json` (project) or `~/.kiro/settings/mcp.json` (home) * Cursor: `.cursor/mcp.json` * Copilot: `.vscode/mcp.json` * Antigravity: `.agents/mcp_config.json` or `~/.gemini/antigravity-cli/mcp_config.json` * Goose: `~/.config/goose/config.yaml` (the `extensions` key) * Copilot CLI: `~/.copilot/mcp-config.json` 2. Lists every MCP server found and its direct command or URL. 3. Reports installed session telemetry hooks. No files are written and no registration happens. **Default table-mode scan may query the Registry when authenticated** to show unregistered names; use `--inventory` for a guaranteed local-only scan. ## Example ```bash observal scan ``` Output: ``` Claude Code (~/.claude/settings.json) filesystem npx @modelcontextprotocol/server-filesystem not wrapped github npx @modelcontextprotocol/server-github not wrapped Kiro (.kiro/settings/mcp.json) mcp-obsidian mcp-obsidian not wrapped 2 harness(s) found, 3 MCP server(s) total, 0 wrapped. ``` ## Scoping to a single harness ```bash observal scan --harness claude-code ``` ## What to do next Once you see what's installed, instrument it: ```bash # Install session telemetry hooks across all harnesses observal doctor patch --all-harnesses # Or target a specific harness observal doctor patch --harness kiro # Preview changes without writing anything observal doctor patch --all-harnesses --dry-run ``` ## Exit codes | Code | Meaning | | ---- | -------------------------------------------------------------------------- | | 0 | Inventory completed (even when empty); default scan found a harness config | | 1 | Unknown harness; default scan found no harness configs | ## Related * [`observal doctor patch`](/cli-reference/doctor): instrument your harnesses (hooks, shims) * [`observal agent pull`](/cli-reference/pull): install a full agent (also wires up MCP servers) * [`observal doctor`](/cli-reference/doctor): diagnose instrumentation end-to-end * [Use Cases -- Observe MCP traffic](/use-cases/observe-mcp-traffic): narrative walkthrough # observal share Create and consume expiring links for version-pinned Agents installed in a repository. Share links preserve Registry authorization: recipients must sign in to the same Observal deployment and can only see and pull Agents available in their current public or teamspace scope. ## Create a share Run the command in a repository and select Agents interactively: ```bash observal share ``` Share every tracked project Agent: ```bash observal share --all --expires-days 7 ``` Select explicit installed Agents: ```bash observal share --agent platform/reviewer --agent platform/github-helper ``` The printed URL is not a web page. Open it only with `observal share open`; the web UI has no share route yet. The bare token also works if the URL's host does not match your configured server (`deployment.frontend_url`). Links expire after 7 days by default. The allowed range is 1 through 30 days. Only project-scoped Agents recorded in the current Registry section of `~/.observal/lockfile.json` are candidates. The repository path, Git remote, harness names, configuration, prompts, and credentials are not uploaded. ## Inspect candidates ```bash observal share candidates observal share candidates --dir ./service --output json ``` Discovery checks every harness section in the lockfile and deduplicates identical Agent/version pairs. ## Open and pull ```bash observal share open 'https://observal.example/shares/agents/' ``` Observal validates that the URL belongs to the configured deployment, fetches the manifest using the configured bearer token, and displays its creator and accessible Agents. It does not contact the host contained in the supplied URL. After selection and confirmation, the command asks for a target harness and delegates each installation to `observal agent pull`. Use `--no-pull` to inspect without installing: ```bash observal share open --no-pull ``` Only currently accessible, approved Agent versions are returned. Inaccessible entries are reported as a count without exposing their identity. ## Revoke The creator or an administrator can revoke a link immediately: ```bash observal share revoke observal share revoke --yes --output json ``` Expired and revoked links cannot be opened. ## Security properties * Public links contain a 256-bit random opaque token, not Agent IDs or repository metadata. * The server stores only a SHA-256 hash of the token. * Links do not grant permissions; Registry visibility is evaluated on every open. * Creation validates every Agent and exact semantic version in one transaction. * URLs with credentials, query strings, fragments, foreign origins, control characters, or invalid paths are rejected. * Pull commands use argument arrays rather than a shell, preventing values from becoming shell syntax. # observal outdated Compare versions recorded in `~/.observal/lockfile.json` with the latest versions available from the active Observal registry. The command never installs an update. ## Synopsis ```bash observal outdated observal outdated --harness claude-code observal outdated --output json --no-report ``` ## Checked items The command checks: * Pulled agents * Separately installed MCP servers * Separately installed skills * Separately installed hooks Components bundled inside a pulled agent are not checked independently. Agent versions pin exact component versions, so update the agent when a newer agent version is reported. The exception is an agent version released before pinning: a component it never locked installs at its latest approved release, with a warning, and `--strict` refuses it. Agent authors use [`observal agent outdated`](/cli-reference/agent#check-component-pins) to see which pinned components have newer releases and which are unlocked. A plain `observal agent pull` keeps an installed agent on its locked version, so the upgrade command for an agent includes `--upgrade`. The command reads only the active registry section of the lockfile. It requires authenticated registry access to retrieve current versions. ## Options | Option | Description | | --------------------- | ------------------------------------------------------------------------------------ | | `--harness ` | Check one registered harness instead of every harness in the active lockfile section | | `--output table` | Render a Rich table; this is the default | | `--output json` | Emit one machine-readable JSON document | | `--report` | Send outdated findings to the user Inbox; this is the default | | `--no-report` | Suppress the Inbox write while still contacting the registry for version checks | Valid harnesses are `cursor`, `kiro`, `claude-code`, `codex`, `copilot`, `copilot-cli`, `opencode`, `antigravity`, `goose`, and `pi`. ## Table output Table output includes every installed item that was checked. Each row has one of these statuses: | Status | Meaning | | ---------- | ----------------------------------------------------------------------------------------------- | | `outdated` | A newer registry version exists | | `current` | The pinned version is current or newer | | `missing` | The pinned item no longer exists in the active registry | | `unknown` | The lockfile has no version for the item; reinstall it with the suggested command to record one | Outdated and unknown rows are followed by a type-specific command using the canonical `namespace/slug` identity. Agents use `observal agent pull --upgrade`; standalone components use their matching registry install command. A missing item is an item-level result, not a command failure, so a completed comparison containing missing rows exits successfully. ## JSON output JSON output has a stable top-level object: ```json { "items": [ { "id": "11111111-1111-4111-8111-111111111111", "qualified_name": "acme/reviewer", "name": "reviewer", "namespace": "acme", "slug": "reviewer", "type": "agent", "harness": "claude-code", "current_version": "1.0.0", "latest_version": "2.0.0", "status": "outdated", "outdated": true, "error": null, "upgrade_command": "observal agent pull acme/reviewer --harness claude-code --no-prompt --upgrade" } ], "summary": { "total": 1, "outdated": 1, "current": 0, "missing": 0, "unknown": 0 }, "report": { "requested": true, "attempted": true, "succeeded": true, "created": 1, "superseded": 0, "error": null } } ``` An empty lockfile returns the same shape with an empty `items` array and zero summary counts. Missing rows include a categorized item-level error with any available request ID. Command-level JSON errors are written to stderr and leave stdout empty. ## Inbox reporting With reporting enabled, outdated findings are sent to the signed-in user's Inbox after the comparison. Duplicate findings are deduplicated by item and latest version. Inbox reporting is best-effort. If the comparison succeeds but reporting fails, the comparison still exits successfully and the `report` object contains the categorized reporting error. Human output prints a warning. Unexpected reporting failures are not suppressed. Use `--no-report` when no Inbox mutation is wanted. This does not make the command offline because registry reads are still required. ## Exit codes | Code | Meaning | | ---- | ------------------------------------------------------------------------------ | | `0` | Comparison completed, including empty, current, outdated, or missing results | | `1` | Unexpected or uncategorized failure, including an unexpected reporting failure | | `2` | Invalid command syntax or output mode | | `3` | Authentication is missing or invalid | | `4` | Registry or filesystem permission denied | | `7` | Invalid harness, malformed lockfile, or invalid installed version | | `8` | Registry rate limit reached | | `9` | Registry unavailable, timed out, or returned invalid version data | | `10` | CLI and server versions are incompatible | ## Related * [`observal agent pull`](/cli-reference/pull): update a pulled agent * [`observal registry`](/cli-reference/registry): install a standalone component * [`observal config`](/cli-reference/config): inspect the active server configuration # observal reconcile Backfill local session records missed by automatic hook or extension delivery. Reconcile is a manual recovery command, not the normal collection path. The server cannot scan files on a developer machine. Automatic hooks and extensions wake the same durable delivery engine during normal harness activity; reconcile scans recent local sources on demand. ## When to use it Run reconcile when: * Telemetry instrumentation was installed after sessions already existed * The machine or server was offline * Delivery was interrupted * Durable outbox records remain pending * Recent local history needs to be verified or backfilled Routine use is unnecessary when automatic delivery is healthy. ## Examples Preview the default seven-day window without network or cursor changes: ```bash observal reconcile --dry-run --output json ``` Backfill every installed harness: ```bash observal reconcile --output json ``` Target one harness and a 24-hour discovery window: ```bash observal reconcile --harness kiro --since 24 --output json ``` The discovery window accepts 1 through 8,760 hours. ## Delivery behavior A non-dry run: 1. Validates configuration and the harness selection before any outbox side effect. 2. Retries the existing durable outbox. 3. Discovers recent session sources through installed harness adapters. 4. Skips locally finalized sources that have not grown. 5. Recovers the contiguous server checkpoint for unfinished sources. 6. Queues and sends only complete records after that checkpoint. 7. Sends final metadata when all records were uploaded but the session was not finalized. 8. Leaves transient failures queued for the next automatic wake-up or reconcile run. 9. Quarantines permanent server rejections and reports them explicitly. Acknowledged checkpoints make repeated runs idempotent. A dry run only reads local sources and cursor state. It does not drain the outbox, contact the ingest API, or update cursor state. ## JSON result ```json { "dry_run": false, "since_hours": 168, "outbox_drained": true, "targets": [ { "harness": "kiro", "discovered": 2, "pushed": 1, "finalized": 0, "queued": 1, "rejected": 0, "would_push": 0, "would_finalize": 0, "up_to_date": 0, "skipped": 0, "errors": 0, "sessions": [ {"session_id": "session-1", "status": "pushed", "bytes_new": 512}, {"session_id": "session-2", "status": "queued", "bytes_new": 128} ] } ], "summary": { "discovered": 2, "pushed": 1, "finalized": 0, "queued": 1, "rejected": 0, "would_push": 0, "would_finalize": 0, "up_to_date": 0, "skipped": 0, "errors": 0 }, "rejections": [] } ``` `outbox_drained: false` means durable records remain pending. A queued session is safely stored for retry and is not reported as delivered. Permanent ingestion failures include their HTTP status in `rejections`. ## Exit codes | Code | Meaning | | ---- | -------------------------------------------------- | | 3 | Session delivery identity is not configured | | 7 | Unknown harness or invalid discovery window | | 9 | Outbox storage or session discovery is unavailable | Per-session source read failures and checkpoint mismatches are explicit result items so other sessions can still be recovered. ## Related * [`observal doctor`](/cli-reference/doctor): configure and verify automatic telemetry instrumentation * [`observal ops telemetry status`](/cli-reference/ops): inspect server and durable outbox health * [Session tracking](/core-concepts/session-tracking): automatic and recovery delivery architecture # observal inbox View and update the signed-in user's work and event feed. Inbox items cover reviews, decisions, teamspace activity, update notices, completed insights, and system notices. The Inbox is self-only. There is no option for reading another user's items. ## List items The direct and explicit list forms are equivalent: ```bash observal inbox --output json observal inbox list --state open --action-required --output json observal inbox list --subject-type mcp --search postgres --sort oldest --output json ``` | Option | Description | | -------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `--state`, `-s` | Filter by `open`, `done`, or `dismissed` | | `--kind`, `-k` | Filter by event kind | | `--action-required` / `--no-action-required` | Filter by whether the item requires action | | `--unread` / `--read` | Filter by read state | | `--subject-type` | Filter by the related object type, such as `agent`, `mcp`, `skill`, `team`, or `insight_report` | | `--search`, `-q` | Search titles, bodies, namespaces, and slugs | | `--sort` | Sort by `newest` or `oldest` | | `--page`, `-p` | Select a page, starting at 1 | | `--page-size` | Return 1 through 100 items | | `--output`, `-o` | Table or JSON output | Supported kinds are: * `review_requested` * `review_approved` * `review_rejected` * `review_comment` * `change_requested` * `team_join_requested` * `team_join_decided` * `team_created_pending` * `ownership_transfer` * `update_available` * `insight_ready` * `system_notice` JSON returns the direct paginated server object: ```json { "items": [], "total": 0, "page": 1, "page_size": 25 } ``` An empty `items` array is successful. Continue with `--page N` while the returned page range is below `total`. ## Count items ```bash observal inbox count --output json observal inbox count --facets --facet-state open --output json ``` The basic result includes `unread`, `action_required`, `open`, `done`, and `dismissed`. `--facets` also returns `by_kind` and `by_subject_type`; `--facet-state` restricts those breakdowns and requires `--facets`. ## Show an item ```bash observal inbox show 11111111-1111-1111-1111-111111111111 --output json ``` The detail result includes the item's body, related subject, exact action URL or command, and append-only history. The CLI displays an action command but never runs it automatically. ## Update one item Every mutation supports table and JSON output: ```bash observal inbox read 11111111-1111-1111-1111-111111111111 --output json observal inbox unread 11111111-1111-1111-1111-111111111111 --output json observal inbox done 11111111-1111-1111-1111-111111111111 --output json observal inbox dismiss 11111111-1111-1111-1111-111111111111 --output json observal inbox reopen 11111111-1111-1111-1111-111111111111 --output json ``` Read state and lifecycle state are separate: * `read` and `unread` change only whether the item has been seen. * `done` resolves completed work. * `dismiss` resolves the item without acting on it. * `reopen` returns a done or dismissed item to `open`. JSON returns the updated item directly. ## Mark a filtered set read ```bash observal inbox read-all --kind update_available observal inbox read-all --state open --subject-type mcp --search postgres --yes --output json ``` `read-all` affects only unread items matching the supplied state, kind, action-required, subject-type, and search filters. Human mode asks for confirmation unless `--yes` is present. JSON mode never prompts and therefore requires `--yes`. JSON returns: ```json { "updated": 3 } ``` ## Exit codes Invalid states, kinds, sort orders, filter lengths, item IDs, and non-interactive confirmation use validation exit code 7. Authentication, permission, not-found, rate-limit, unavailable-service, and version failures preserve the shared CLI exit contract. In JSON mode, stdout contains only the successful result. Failures leave stdout empty and write one categorized error object to stderr. ## Related * [`observal outdated`](/cli-reference/outdated): report installed updates to Inbox * [`observal registry`](/cli-reference/registry): inspect Registry subjects referenced by Inbox items # observal agent pull Install a complete Agent into a harness. Pull resolves the Agent version to install, asks the server for harness-native config built from the exact component versions that Agent version pinned, merges generated files safely, installs bundled skills and hooks, runs required harness setup, and records exact installed state in `observal.lock` and the local lockfile. ## Synopsis ```bash observal agent pull --harness [OPTIONS] ``` Agent references may be UUIDs, canonical `namespace/slug`, unambiguous bare names, aliases, or row numbers from the latest Agent list. ## Examples ```bash observal agent pull alice/reviewer --harness kiro --no-prompt --output json observal agent pull alice/reviewer --harness claude-code --scope project --dry-run --no-prompt --output json observal agent pull alice/reviewer --harness pi --version 1.2.3 --no-prompt --output json observal agent pull alice/reviewer --harness cursor --no-prompt --upgrade observal agent pull alice/reviewer --harness cursor --no-prompt --strict ``` ## Options | Option | Description | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `--harness`, `-i` | Required target: `cursor`, `kiro`, `claude-code`, `codex`, `copilot`, `copilot-cli`, `opencode`, `antigravity`, `goose`, or `pi` | | `--dir`, `-d` | Project directory used to resolve generated paths | | `--dry-run`, `-n` | Return planned files and setup commands without changing disk or installation metadata | | `--scope` | `project` or `user`, only for harnesses that support explicit scope | | `--model` | Model ID, or `harness=model`; repeatable | | `--tools` | Claude Code tool allowlist | | `--refresh-models` | Refresh the model catalog before an interactive model picker | | `--no-prompt`, `-y` | Disable environment, header, scope, and model prompts | | `--env`, `-e` | MCP environment value in `NAME=VALUE` form; repeatable | | `--header`, `-H` | MCP header in `NAME=VALUE` form; repeatable | | `--version`, `-V` | Install this exact Agent version and lock it | | `--upgrade` | Install the latest approved Agent version instead of the locked one, and lock it | | `--strict` / `--no-strict` | Refuse an install that does not match its lock; defaults to `OBSERVAL_STRICT` | | `--output`, `-o` | Table or JSON output | Unknown harnesses, unsupported scopes, malformed assignments, unused harness model overrides, unsupported model or tool options, invalid versions, and `--upgrade` combined with `--version` fail locally with validation exit code 7. ## Pinned versions Pulls are pinned at two levels. **Components.** Every Agent version pins each of its MCP servers, skills, hooks, prompts, and sandboxes to one exact component version, identified by its version id and a content digest. Pull always installs those pinned versions. A component that ships a newer version never changes what an existing Agent version installs; the Agent's author releases a new Agent version to move it (see [`observal agent release --refresh-components`](/cli-reference/agent#release-and-versions)). **The Agent version.** Pull chooses the Agent version in this order: 1. `--version X`: exactly that version. 2. `--upgrade`: the latest approved version. 3. `observal.lock` in `--dir`: the version the project locked. 4. `~/.observal/lockfile.json`: the version this machine already installed for this harness and directory. 5. Otherwise, a first install: the latest approved version. A plain pull therefore keeps an existing install on its version after newer versions are approved, and says when one is available. Only `--upgrade` or `--version` moves it. ### `observal.lock` Project-scope pulls write `observal.lock` to `--dir`. Commit it: teammates and CI pulling the same Agent in that project install the same Agent version, and so the same component versions, until someone runs `--upgrade` or `--version` and commits the change. ```json { "lock_version": 1, "agents": { "alice/reviewer": { "id": "11111111-1111-1111-1111-111111111111", "version": "1.2.3", "lock_digest": "sha256:…", "components": [ {"type": "mcp", "qualified_name": "acme/github", "version": "1.4.2", "digest": "sha256:…"} ] } } } ``` Entries are keyed by `namespace/slug` and carry the Agent's registry `id`, so an Agent that was renamed or transferred keeps its locked version; the next pull rewrites the entry under the new name. User-scope installs are not tied to a project and neither read nor write `observal.lock`. `--dry-run` never writes it. A malformed or unsupported `observal.lock` fails with validation exit code 7 before anything is installed. If `observal.lock` or the local lockfile names an Agent version this server does not have, pull fails with not-found exit code 5 and says which lock pinned it; `--upgrade` or `--version` moves past it and rewrites the lock. ### Strict mode Without strict mode, pull installs and warns when: * a component of an Agent version released before pinning has no lock, so its latest version is installed; * a pinned component version no longer matches the digest recorded when the Agent version was locked; * a pinned component version is not approved; * the Agent version no longer matches the lock digest recorded in `observal.lock`. `--strict`, or `OBSERVAL_STRICT=1` for CI, turns each of these into a conflict (exit code 6) before any file is written. The flag wins over the environment variable, so `--no-strict` is the escape hatch in a strict pipeline. A server that predates component locks cannot check any of this, so a strict pull against it fails with version-mismatch exit code 10 instead of installing unchecked. JSON mode cannot prompt and requires `--no-prompt`. Missing required component values fail before config generation with `error.result.needs_input: true` and a list of names and component labels. ## Secrets Pull discovers required MCP environment variables and headers from the Agent's components. Interactive mode prompts for missing values. Non-interactive mode uses matching `--env` and `--header` assignments and stops before installation when any required value is missing. Optional values may remain unset. Values are sent only in the installation request and generated config. They are not included in JSON results, success messages, traces, or error details. Prefer environment expansion or secure shell input so secrets do not remain in shell history. ## File safety Generated relative paths are confined to `--dir`. Home paths are allowed only for an explicit user-scope installation supported by that harness. Absolute paths, parent traversal, and symlink escapes are rejected before installation tracking is updated. Pull behavior by file type: * JSON MCP and hook sections merge into existing objects. * YAML sections merge only when the existing top level and target section are mappings. * TOML managed tables are replaced idempotently while unrelated tables remain. * Generated text, prompt, Agent, and hook config files use atomic replacement. * Malformed or structurally incompatible existing config is never overwritten. The command exits with conflict code 6 and leaves it untouched. No `OTEL_*` or harness telemetry environment variables are generated. Session telemetry continues through Observal-managed hooks and reconciliation. ## Installation sequence Pull performs these steps: 1. Validate harness, scope, model, tool, assignment, version, and output combinations. 2. Resolve the canonical Agent, collect install options, and choose the Agent version (see [Pinned versions](#pinned-versions)). 3. Load MCP environment and header requirements from the pinned component versions. 4. Check installed component version conflicts. 5. Request the harness-specific installation config and its lock; refuse here in strict mode. 6. Resolve and validate every generated path. 7. Write or preview files and install bundled skills. 8. Run required harness MCP registration commands. 9. Record the installed Agent and component versions in the Registry-scoped lockfile and, for project-scope installs, in `observal.lock`. 10. Refresh the local layer snapshot and active-Agent state. Failed skill installation or MCP setup prevents installation metadata from being recorded. Setup commands have a 60-second timeout; a timeout is reported as a setup failure with exit code 9. A lockfile write failure is also reported as exit code 9 instead of claiming success. Failures after filesystem changes include safe partial state under `error.result`, including the stage, written file statuses, setup executable status, and tracking state. Setup arguments and secret values are omitted. A layer-snapshot failure is returned as a visible warning because the generated harness installation remains usable. ## JSON result Successful JSON output has this shape: ```json { "agent": { "id": "11111111-1111-1111-1111-111111111111", "qualified_name": "alice/reviewer", "version": "1.2.3", "latest_version": "1.3.0", "resolved_from": "project-lock", "local_name": "reviewer" }, "project_lock": "/work/project/observal.lock", "lock": { "status": "locked", "digest": "sha256:…", "components": [ { "type": "mcp", "name": "GitHub", "id": "22222222-2222-2222-2222-222222222222", "version": "1.4.2", "version_id": "33333333-3333-3333-3333-333333333333", "digest": "sha256:…", "qualified_name": "acme/github", "source": "lock" } ], "problems": [] }, "harness": "kiro", "scope": "project", "dry_run": false, "target_directory": "/work/project", "files": [ { "path": "/work/project/.kiro/agents/reviewer.json", "status": "created" } ], "warnings": [], "setup_commands": [], "reports_sessions": true } ``` File statuses include `created`, `updated`, `merged`, `installed`, `cloned`, `would write`, and `would clone`. `agent.version` is the version that was installed and `agent.resolved_from` says why: `requested`, `upgrade`, `project-lock`, `installed`, or `latest`. `lock.status` is `locked`, `partial`, or `unlocked`; each component's `source` is `lock`, `version` (matched by its recorded version string), or `fallback-latest`. `lock.problems` lists what strict mode would refuse. `project_lock` is null for user-scope installs and dry runs. `reports_sessions` is true when written hook files contain Observal session push commands, including hooks retained during a merge. These hooks can report prompts, tool calls, and tool output to the configured server when they run, whether or not `observal doctor patch` was run. It does not guarantee successful delivery. On a failed pull, check `error.result.reports_sessions` when available: a hook file may already be on disk and active even when this pull wrote no files and the Agent was not recorded as installed. Dry-run returns the same shape with `dry_run: true`, planned statuses, and `would_run` setup actions. In dry-run, `reports_sessions` predicts whether session hooks **would** be present after applying the plan; it does not mean the preview installed them. Dry-run does not write files, execute setup commands, update the lockfile or `observal.lock`, persist an active Agent, or emit a pull audit event. ## Human output Human mode lists every created, updated, merged, installed, cloned, or planned path. Component version conflicts, server warnings, snapshot warnings, and setup commands are printed explicitly. When session push hooks are configured, a telemetry line names the server they can report to. ## Exit codes | Code | Meaning | | ---- | ------------------------------------------------------------------------------------------------- | | 3 | Authentication required or failed | | 4 | Agent or component access denied | | 5 | Agent or component not found | | 6 | Existing config cannot be merged safely, or a strict install does not match its lock | | 7 | Invalid harness, scope, version, path, assignment, or option combination | | 8 | Rate limit reached | | 9 | Server, filesystem, skill source, lockfile, or setup command unavailable | | 10 | CLI and server version mismatch, including a strict pull against a server without component locks | ## Related * [`observal agent`](/cli-reference/agent): create and publish Agents * [`observal scan`](/cli-reference/scan): inspect installed harness content * [`observal outdated`](/cli-reference/outdated): compare installed Agent versions * [`observal agent outdated`](/cli-reference/agent#check-component-pins): see which components an Agent version pins behind their latest release * [`observal doctor`](/cli-reference/doctor): verify hooks and local installation state # observal registry Publish and manage registry components. The registry has five component types: MCP servers, skills, hooks, prompts, and sandboxes. ## Subcommand structure ``` observal registry [args] ``` | Type | Submit | List | My | Show | Install | Render | Edit | | --------- | ------ | ---- | --- | ---- | ------- | ------ | ---- | | `mcp` | yes | yes | yes | yes | yes | no | yes | | `skill` | yes | yes | yes | yes | yes | no | yes | | `hook` | yes | yes | no | yes | yes | no | yes | | `prompt` | yes | yes | yes | yes | no | yes | yes | | `sandbox` | yes | yes | no | yes | no | no | yes | Every component type also supports archive, unarchive, ownership transfer, and co-author management. Registry also contains the `models`, `version`, `recommend`, and mixed `bulk` groups. All registry references accept a UUID, canonical `namespace/slug`, a unique legacy bare name, a row number from the latest human list output for the same component type, or an `@alias`. Agents and scripts must use returned UUIDs or `qualified_name` values, never row numbers. If the same bare slug exists in multiple namespaces, qualify it, for example `alice/search` instead of `search`. ### Shared lifecycle and collaboration commands ```bash observal registry skill archive alice/reviewer --yes --output json observal registry skill unarchive alice/reviewer --yes --output json observal registry skill transfer-owner alice/reviewer bob --yes --output json observal registry skill co-authors list alice/reviewer --output json observal registry skill co-authors add alice/reviewer bob@example.com --output json observal registry skill co-authors remove alice/reviewer --output json ``` Archive, restore, and ownership transfer require explicit confirmation in JSON mode. Their JSON output is the direct server result. Co-author list returns the standard list envelope; add and remove return the direct server result. The namespace is the publisher's username or a teamspace handle. Usernames cannot change after the account owns a registry listing. Team members can browse approved private teamspace items in normal list results. Use `--team TEAM_HANDLE` to include public items plus that team's private items, or `--namespace TEAM_HANDLE` to restrict results to that namespace. Direct references use `team-handle/item-slug`. Nonmembers receive the same not-found response for private items as for unknown items. ### Teamspace visibility Use the teamspace target and visibility options on submit commands: ```bash observal registry skill submit --skill-md ./SKILL.md --team platform-tools --visibility public observal registry skill submit --skill-md ./SKILL.md --team platform-tools --visibility team observal registry skill list --team platform-tools --search 'frontend design' --harness claude-code --output json observal registry skill list --namespace platform-tools --output json observal registry skill show platform-tools/internal-skill --output json ``` `public` teamspace items are visible to all registry users. `team` items are visible only to team members and privileged reviewers. Team owners and reviewers can change visibility after publication. A team member's new submission still follows the normal review workflow. *** ## Mixed bulk submission Submit up to 200 MCP, skill, hook, prompt, and sandbox entries from one JSON file: ```bash observal registry bulk submit --from-file components.json --dry-run --output json observal registry bulk submit --from-file components.json --yes --output json ``` The file is a bare array or an object with a `components` array. Each entry contains `type` plus the normal API submission fields: ```json { "components": [ { "type": "skill", "name": "review-helper", "version": "1.0.0", "description": "Reviews changes", "owner": "alice", "task_type": "code-review", "delivery_mode": "registry_direct", "skill_md_content": "---\nname: review-helper\ndescription: Reviews changes\n---\n" }, { "type": "prompt", "name": "review-brief", "description": "Review prompt", "owner": "alice", "category": "general", "template": "Review {{change}}" } ] } ``` Dry run validates file structure without contacting submission endpoints. Execution structurally validates every entry before the first mutation, submits entries in order, reports conflicts as skipped, and returns per-entry IDs, canonical names, review status, and safe errors. Results include `partial`; item errors set it to `true` and exit with code 11, while conflicts counted as skips remain successful. Authentication, permission, rate-limit, version, and service failures stop the batch. JSON execution requires `--yes`. Re-running a partially completed file is safe only after inspecting results. Existing identities are skipped for component types that reject duplicates. Verify created items by returned UUID or `qualified_name`. *** ## MCP servers MCP server registry commands for submitting, browsing, generating configuration, editing, and archiving MCP server listings. ### `observal registry mcp submit` Submit an MCP server to the registry. By default, paste your server's JSON config (the same format you use in your harness). Use `--git` to analyze a git repository instead. #### Synopsis ```bash observal registry mcp submit [OPTIONS] observal registry mcp submit --git [OPTIONS] ``` #### Options | Option | Short | Description | | ------------ | ----- | -------------------------------------------------- | | `--git` | `-g` | Analyze a git repository instead of pasting config | | `--name` | `-n` | Pre-fill server name (skip prompt) | | `--category` | `-c` | Pre-fill category (skip prompt) | | `--yes` | `-y` | Accept all defaults | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit an existing draft for review (MCP ID) | | `--output` | `-o` | Output format: `table` or `json` | #### Default flow (JSON paste) 1. Prompts you to paste your MCP server JSON config. 2. Accepts multiple formats: * **harness config**: `{"mcpServers": {"name": {"command": "...", "args": [...], "env": {...}}}}` * **Bare config**: `{"command": "npx", "args": ["-y", "pkg"]}` * **SSE/HTTP**: `{"url": "http://...", "type": "sse", "headers": {...}}` * **server.json manifest**: `{"packages": [...], "remotes": [...]}` 3. Auto-detects environment variables from `$VAR` patterns and `env` keys. 4. Shows a config preview and prompts for metadata (name, description, category). 5. Submits to registry for review. #### Git analysis flow (`--git`) 1. Shallow-clones the repo. 2. Detects the MCP framework (FastMCP, MCP SDK, TypeScript SDK, Go SDK). 3. Extracts server name, description, and exposed tools via AST. 4. Scans for required env vars (`os.environ`, `os.getenv`, `.env.example`, `server.json`). 5. Prompts for metadata confirmation. 6. Submits to registry for review. #### Examples ```bash # Paste config (default, recommended) observal registry mcp submit # Non-interactive with piped JSON echo '{"command": "npx", "args": ["-y", "@example/mcp-server"]}' | observal registry mcp submit -y -n my-server -c developer-tools --output json # Save as draft observal registry mcp submit --draft # Analyze a git repo observal registry mcp submit --git https://github.com/MarkusPfundstein/mcp-obsidian # Non-interactive git analysis observal registry mcp submit --git https://github.com/sooperset/mcp-atlassian -y # Submit an existing draft for review observal registry mcp submit --submit my-server ``` #### Valid categories `browser-automation`, `cloud-platforms`, `code-execution`, `communication`, `databases`, `developer-tools`, `devops`, `file-systems`, `finance`, `knowledge-memory`, `monitoring`, `multimedia`, `productivity`, `search`, `security`, `version-control`, `ai-ml`, `data-analytics`, `general`. #### Valid transports `stdio`, `sse`, `streamable-http`. #### Valid frameworks `python`, `docker`, `typescript`, `go`. *** ### `observal registry mcp list` List approved MCP servers in the registry. ```bash observal registry mcp list [--search TERM] [--category CAT] [--limit N] [--sort name|category|version] [--output table|json] [--interactive] ``` | Option | Short | Description | | --------------- | ----- | -------------------------------------- | | `--search` | `-s` | Search by name or description | | `--category` | `-c` | Filter by category | | `--limit` | `-n` | Max results (default: 50) | | `--sort` | | Sort by: `name`, `category`, `version` | | `--output` | `-o` | Output format: `table`, `json` | | `--interactive` | `-i` | Open a fuzzy-search picker | ```bash observal registry mcp list --search github observal registry mcp list --category ai-ml --output json observal registry mcp list --interactive observal registry mcp list --sort category --limit 10 ``` *** ### `observal registry mcp my` List your own MCP servers across all statuses (draft, pending, approved, rejected). ```bash observal registry mcp my [--output table|json] ``` ```bash observal registry mcp my observal registry mcp my --output json ``` *** ### `observal registry mcp show` Show full details of an MCP server including validation results, env vars, and supported harnesses. ```bash observal registry mcp show [--output table|json] ``` ```bash observal registry mcp show my-server observal registry mcp show 3 observal registry mcp show @fav --output json ``` *** ### `observal registry mcp install` Generate a harness config snippet for an MCP server. This command does not write harness configuration or record an installation. Interactive mode prompts for required environment variables and headers. JSON mode requires `--no-prompt` and rejects missing required values before config generation with `error.result.needs_input: true`. ```bash observal registry mcp install --harness [options] ``` | Option | Short | Description | | ------------- | ----- | -------------------------------------------------------------------------------- | | `--harness` | `-i` | Target harness (required) | | `--version` | `-V` | Generate configuration from that exact version instead of the latest | | `--env` | `-e` | Environment value as `KEY=VALUE`; repeatable | | `--header` | | Header value as `KEY=VALUE`; repeatable | | `--env-file` | | Read environment values from a file | | `--no-prompt` | `-y` | Use supplied values without prompting; JSON fails if required values are missing | | `--raw` | | Output only a bare config template; missing values remain explicit placeholders | | `--output` | `-o` | Output the complete operation result as table or JSON | ```bash observal registry mcp install my-server --harness claude-code observal registry mcp install my-server --harness cursor --raw > .cursor/mcp.json observal registry mcp install 2 --harness copilot observal registry mcp install @db --harness kiro ``` *** ### `observal registry mcp edit` Edit an MCP server submission. For draft/pending/rejected listings, edits in place. For approved listings, publishes a new version with a semver bump. ```bash observal registry mcp edit [OPTIONS] ``` | Option | Short | Description | | --------------- | ----- | ---------------------------------------------------------------- | | `--from-file` | `-f` | Load updates from a JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--category` | `-c` | New category | | `--version` | `-v` | New version string | | `--git-url` | | New git URL | | `--command` | | New command | | `--url` | | New URL (SSE/HTTP) | | `--bump` | | Version bump for approved listings: `patch`, `minor`, or `major` | | `--changelog` | | Changelog for an approved-listing version | | `--output` | `-o` | Output format: `table` or `json` | Without flags, opens an interactive JSON paste prompt (same format as submit). ```bash # Interactive JSON paste edit observal registry mcp edit my-server # Update specific fields observal registry mcp edit my-server -d "New description" -c databases # Load updates from a file observal registry mcp edit my-server --from-file updates.json # Bump version on an approved listing observal registry mcp edit my-server --version 1.2.0 ``` *** ### `observal registry mcp transfer-owner` Transfer ownership to another username. You stop being the owner immediately. ```bash observal registry mcp transfer-owner my-server @alice -y ``` *** ## Skills Skill registry commands. Skills are portable SKILL.md instruction packages that provide agents with task-specific guidance. Valid task types: `code-review`, `code-generation`, `testing`, `documentation`, `debugging`, `refactoring`, `deployment`, `security-audit`, `performance`, `general`. ### `observal registry skill submit` Submit a new skill for review. Provide `--git-url` to let the server fetch SKILL.md automatically, or use `--skill-md` to paste content directly. ```bash observal registry skill submit [OPTIONS] ``` | Option | Short | Description | | ------------- | ----- | ----------------------------------------------------- | | `--from-file` | `-f` | Create from JSON file | | `--skill-md` | | Path to SKILL.md (auto-fills fields from frontmatter) | | `--git-url` | | Git repository URL | | `--git-ref` | | Branch or tag (default: main) | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (skill ID) | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry skill submit --git-url https://github.com/org/repo observal registry skill submit --from-file skill.json observal registry skill submit --skill-md ./SKILL.md --git-url https://github.com/org/repo --name review --description "Review code" --task-type code-review --output json observal registry skill submit --draft observal registry skill submit --submit abc123 ``` *** ### `observal registry skill list` List approved skills in the registry. ```bash observal registry skill list [--task-type TYPE] [--target-agent AGENT] [--search TERM] [--output table|json] ``` | Option | Short | Description | | ---------------- | ----- | ------------------------------ | | `--task-type` | `-t` | Filter by task type | | `--target-agent` | | Filter by target agent | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash observal registry skill list observal registry skill list --task-type code-review observal registry skill list --target-agent claude-code --output json observal registry skill list --search "refactor" ``` *** ### `observal registry skill my` List your own skills across all statuses (draft, pending, approved, rejected). ```bash observal registry skill my [--output table|json] ``` ```bash observal registry skill my observal registry skill my --output json ``` *** ### `observal registry skill show` Show detailed information about a skill, including validation status, task type, git source, and slash command. ```bash observal registry skill show [--output table|json] ``` ```bash observal registry skill show my-skill observal registry skill show 1 observal registry skill show @refactor-skill --output json ``` *** ### `observal registry skill install` Install a skill by fetching the full skill directory from git. Clones the skill directory via sparse checkout and writes it to the appropriate harness skill path. ```bash observal registry skill install --harness [--scope user|project] [--raw] [--no-write] ``` | Option | Short | Description | | ------------ | ----- | ------------------------------------------------------------- | | `--harness` | `-i` | Target harness (required) | | `--scope` | `-s` | Install scope: `user` (global, default) or `project` | | `--raw` | | Output raw JSON only | | `--no-write` | | Generate config without writing skill files or lockfile state | | `--version` | `-V` | Install one version instead of the latest | | `--output` | `-o` | Output the operation result as table or JSON | Scopes: * `user` (default): writes to `~/./skills//` globally. * `project`: writes to `.agents/skills//` in the current directory, then symlinks into detected harness config directories. JSON output does not disable installation. It returns whether files were written and the installed path. Raw and no-write modes do not record the skill as installed. An installed skill is recorded with its exact version, version id, and digest, with or without `--version`. A failed file write or lockfile update returns a categorized failure instead of reporting success. ```bash observal registry skill install my-skill --harness claude-code observal registry skill install @sk --harness kiro --scope project observal registry skill install 2 --harness cursor --raw observal registry skill install my-skill --harness antigravity --no-write ``` *** ### `observal registry skill edit` Edit a draft, pending, or rejected skill submission. Acquires an edit lock to prevent concurrent modifications. ```bash observal registry skill edit [OPTIONS] ``` | Option | Short | Description | | --------------- | ----- | -------------------------------- | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--task-type` | `-t` | New task type | | `--git-url` | | New git URL | | `--git-ref` | | New git ref | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry skill edit my-skill --description "Better desc" observal registry skill edit abc123 --from-file updates.json observal registry skill edit @sk --git-url https://github.com/org/new-repo observal registry skill edit 2 --version 2.0.0 --task-type debugging ``` *** ### `observal registry hook submit` Submit a new hook for review. Supports inline script content via `--script`, or git-hosted hooks via `--source-url`. ```bash observal registry hook submit [OPTIONS] ``` | Option | Short | Description | | --------------- | ----- | ----------------------------------------------------- | | `--from-file` | `-f` | Create from JSON file | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (hook ID) | | `--script` | | Path to hook script file (content stored in registry) | | `--source-url` | | Git repo containing hook scripts | | `--source-ref` | | Branch/tag to track (default: main) | | `--source-path` | | Directory within repo containing hook files | | `--requires` | | Install prerequisites (repeatable) | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry hook submit observal registry hook submit --script ./protect-files.sh observal registry hook submit --source-url https://github.com/org/hooks --source-path hooks/guard/ observal registry hook submit --from-file hook.json observal registry hook submit --draft observal registry hook submit --submit abc123 ``` *** ### `observal registry hook list` List approved hooks from the registry. ```bash observal registry hook list [--event EVENT] [--search TERM] [--output table|json] ``` | Option | Short | Description | | ---------- | ----- | ------------------------------ | | `--event` | `-e` | Filter by event type | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash observal registry hook list observal registry hook list --event Stop observal registry hook list --search guard --output json ``` *** ### `observal registry hook show` Show detailed information for a single hook, including event type, handler config, and execution mode. ```bash observal registry hook show [--output table|json] ``` ```bash observal registry hook show my-hook observal registry hook show 1 observal registry hook show @guard --output json ``` *** ### `observal registry hook install` Install a hook for a specific harness. Writes script files and merges hook config into the harness's settings. Existing hooks are preserved during merge. ```bash observal registry hook install --harness [--platform PLATFORM] [--raw] [--dir DIR] [--version VERSION] ``` | Option | Short | Description | | ------------ | ----- | -------------------------------------------------------- | | `--harness` | `-i` | Target harness (required) | | `--platform` | `-p` | Platform: `win32`, `darwin`, `linux` | | `--raw` | | Output raw JSON only (no file writes) | | `--dir` | `-d` | Project directory for file writes (default: cwd) | | `--version` | `-V` | Install one version instead of the latest | | `--output` | `-o` | Output the complete installation result as table or JSON | ```bash observal registry hook install my-hook --harness claude-code observal registry hook install @guard --harness kiro --dir ./project observal registry hook install my-hook --harness cursor --raw observal registry hook install my-hook --harness claude-code --platform darwin observal registry hook install my-hook --harness claude-code --output json ``` Hook installation validates every path before writing, refuses to replace malformed existing JSON, writes files atomically, and does not duplicate an existing event entry when repeated. The lockfile records the installed version, its version id and digest, and any version you requested. *** ### `observal registry hook edit` Edit a draft, pending, or rejected hook submission. Acquires an edit lock to prevent concurrent modifications. ```bash observal registry hook edit [OPTIONS] ``` | Option | Short | Description | | --------------- | ----- | -------------------------------- | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--event` | `-e` | New event type | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry hook edit my-hook --description "Updated guard hook" observal registry hook edit my-hook --event Stop --version 1.1.0 observal registry hook edit @guard --from-file updated-hook.json observal registry hook edit 1 --name new-name ``` *** ### `observal registry prompt submit` Submit a new prompt template for review. You can submit interactively, from a JSON file, or from a raw template file. ```bash observal registry prompt submit [OPTIONS] ``` | Option | Short | Description | | ------------- | ----- | -------------------------------------------------------- | | `--from-file` | `-f` | Create from JSON file, or read template from a text file | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (prompt ID) | | `--output` | `-o` | Output format: `table` or `json` | If `--from-file` points to a non-JSON file, its content is used as the template and you are prompted for metadata interactively. ```bash observal registry prompt submit observal registry prompt submit --from-file prompt.json observal registry prompt submit --from-file template.md observal registry prompt submit --draft observal registry prompt submit --submit abc123 ``` *** ### `observal registry prompt list` List approved prompts in the registry. ```bash observal registry prompt list [--category CAT] [--search TERM] [--output table|json] ``` | Option | Short | Description | | ------------ | ----- | ------------------------------ | | `--category` | `-c` | Filter by category | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash observal registry prompt list observal registry prompt list --category code-review observal registry prompt list --search "refactor" --output json ``` *** ### `observal registry prompt my` List your own prompts across all statuses (draft, pending, approved, rejected). ```bash observal registry prompt my [--output table|json] ``` ```bash observal registry prompt my observal registry prompt my --output json ``` *** ### `observal registry prompt show` Show detailed information about a prompt, including the template content. ```bash observal registry prompt show [--output table|json] ``` ```bash observal registry prompt show my-prompt observal registry prompt show 1 observal registry prompt show @refactor-prompt --output json ``` *** ### `observal registry prompt render` Render a prompt template with variable substitution. Sends key=value pairs to the server, which substitutes them into the template and returns the rendered output. ```bash observal registry prompt render --var key=value [--var key2=value2 ...] ``` | Option | Short | Description | | ---------- | ----- | ------------------------------------ | | `--var` | `-v` | Variable as `key=value` (repeatable) | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry prompt render my-prompt --var lang=python observal registry prompt render @tpl --var file=main.py --var task=refactor ``` *** ### `observal registry prompt edit` Edit a draft, pending, or rejected prompt submission. Acquires an edit lock to prevent concurrent modifications. ```bash observal registry prompt edit [OPTIONS] ``` | Option | Short | Description | | --------------- | ----- | -------------------------------- | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--category` | `-c` | New category | | `--template` | `-t` | New template text | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry prompt edit my-prompt --description "Updated desc" observal registry prompt edit abc123 --from-file updates.json observal registry prompt edit @tpl --template "New template: {{ var }}" observal registry prompt edit 2 --version 2.0.0 --category debugging ``` *** ### `observal registry sandbox submit` Submit a new sandbox environment for review. ```bash observal registry sandbox submit [OPTIONS] ``` | Option | Short | Description | | ------------- | ----- | ---------------------------------------------- | | `--from-file` | `-f` | Create from JSON file | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (sandbox ID) | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry sandbox submit observal registry sandbox submit --from-file sandbox.json observal registry sandbox submit --draft observal registry sandbox submit --submit abc123 ``` *** ### `observal registry sandbox list` List approved sandboxes in the registry. ```bash observal registry sandbox list [--runtime TYPE] [--search TERM] [--output table|json] ``` | Option | Short | Description | | ----------- | ----- | ------------------------------ | | `--runtime` | `-r` | Filter by runtime type | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash observal registry sandbox list observal registry sandbox list --runtime docker observal registry sandbox list --search "node" --output json ``` *** ### `observal registry sandbox show` Show detailed information about a sandbox, including runtime type, container image, and resource limits. ```bash observal registry sandbox show [--output table|json] ``` ```bash observal registry sandbox show my-sandbox observal registry sandbox show 1 observal registry sandbox show @dev-env --output json ``` *** Sandboxes are attached to agents by UUID and are installed when the agent is pulled. There is no standalone Sandbox install command. ```bash observal agent add sandbox observal agent build ``` *** ### `observal registry sandbox edit` Edit a draft, pending, or rejected sandbox submission. Acquires an edit lock to prevent concurrent modifications. ```bash observal registry sandbox edit [OPTIONS] ``` | Option | Short | Description | | ---------------- | ----- | -------------------------------- | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--runtime-type` | `-r` | New runtime type | | `--image` | `-i` | New container image | | `--output` | `-o` | Output format: `table` or `json` | ```bash observal registry sandbox edit my-sandbox --image node:20-alpine observal registry sandbox edit abc123 --from-file updates.json observal registry sandbox edit @env --runtime-type docker --version 2.0.0 --output json ``` *** ## Component versions Use `observal registry version publish` and `observal registry version list` for all five component types. Publication supports direct JSON results; history supports explicit pagination. See [`observal registry version`](/cli-reference/component) for the complete contract. ## Personalized recommendations Use `observal registry recommend` to rank visible components against the signed-in user's sessions and to dismiss or mark recommendations as installed. See [`observal registry recommend`](/cli-reference/recommend) for the JSON schema and feedback actions. # observal registry recommend List Registry components ranked for the signed-in user and record recommendation feedback. ## List The direct and explicit list forms are equivalent: ```bash observal registry recommend --output json observal registry recommend list --limit 12 --type mcp --refresh --output json ``` | Option | Description | | ---------------- | ----------------------------------------------------- | | `--limit`, `-n` | Number of recommendations, from 1 through 24 | | `--type`, `-t` | Restrict to MCP, Skill, Hook, Prompt, or Sandbox | | `--refresh` | Recompute the work profile instead of using its cache | | `--output`, `-o` | Table or JSON output | JSON returns the direct server object. Important fields include: | Field | Meaning | | -------------------- | ---------------------------------------- | | `personalized` | Whether ranking used the user's sessions | | `profile_sessions` | Session count behind the profile | | `topics` | Dominant detected work topics | | `items` | Recommended visible components | | `items[].reason` | Server-provided explanation | | `items[].matched_on` | Terms from the user's own work profile | An empty `items` array is a successful result. It means either no visible components remain or everything applicable has already been installed or dismissed. When `personalized` is false, results are popularity-based and must not be described as tailored. ## Dismiss ```bash observal registry recommend dismiss skill acme/reviewer --output json observal registry recommend dismiss mcp acme/postgres --action not_relevant --output json observal registry recommend dismiss hook acme/guard --action installed --output json ``` Valid actions are `dismissed`, `not_relevant`, and `installed`. JSON returns: ```json { "component_type": "mcp", "component_id": "498c17ac-1234-4567-89ab-cdef01234567", "action": "installed" } ``` Dismissals are scoped to the signed-in user. The CLI cannot inspect another user's recommendation profile. ## Exit codes Invalid component types and actions use validation exit code 7. Authentication, permission, rate-limit, unavailable-service, and version-mismatch failures preserve the shared CLI exit contract. JSON failures leave stdout empty and write one error object to stderr. ## Related * [`observal registry`](/cli-reference/registry): browse components directly * [`observal outdated`](/cli-reference/outdated): check installed versions # observal registry version Publish and inspect versioned releases for MCP servers, skills, hooks, prompts, and sandboxes. ## Commands | Command | Description | | --------- | ------------------------------------------ | | `publish` | Publish a new component version for review | | `list` | List paginated version history | Valid component types are `mcp`, `skill`, `hook`, `prompt`, and `sandbox`. ## Publish ```bash observal registry version publish mcp acme/server \ --version 2.0.0 \ --description "New authentication flow" \ --changelog "Breaking change" \ --output json ``` | Option | Description | | --------------------- | --------------------------------------- | | `--version`, `-v` | Version to publish | | `--description`, `-d` | Required release description | | `--changelog` | Optional release notes | | `--harness` | Supported harness; repeatable | | `--extra` | Type-specific metadata as a JSON object | | `--output`, `-o` | Table or JSON output | Human mode fetches version suggestions and prompts when a version is omitted. JSON mode never prompts and therefore requires an explicit version. Suggestion failures retain their original authentication, rate-limit, unavailable, or version-mismatch category. The extra value must be a JSON object. Versions and harnesses are validated before publication. ## List ```bash observal registry version list mcp acme/server --output json observal registry version list hook acme/guard --page 2 --page-size 100 --output json ``` | Option | Description | | ---------------- | ---------------------------------- | | `--page` | Page number, starting at 1 | | `--page-size` | Items per page, from 1 through 200 | | `--output`, `-o` | Table or JSON output | JSON returns the direct paginated server object with `items`, `total`, `page`, and `page_size`. Empty pages preserve that shape. Human output shows version, review status, relative release date, and publisher. ## Exit codes | Code | Meaning | | ---- | ----------------------------------------------------- | | `0` | Publication or listing completed | | `2` | Invalid command syntax or pagination bounds | | `3` | Authentication required or failed | | `4` | Permission denied | | `5` | Component not found | | `6` | Version or state conflict | | `7` | Invalid component type, version, harness, or metadata | | `8` | Rate limit reached | | `9` | Registry unavailable | | `10` | CLI and server version mismatch | ## Related * [`observal registry`](/cli-reference/registry): component management * [`observal outdated`](/cli-reference/outdated): compare installed and latest versions # observal registry models Display the model catalog packaged for registered harnesses. ## Synopsis ```bash observal registry models [--harness ] [--output table|json] observal registry models list [--harness ] [--output table|json] ``` The direct and explicit list forms are equivalent. ## Options | Option | Description | | ------------------ | -------------------------------------- | | `--harness ` | Filter to one registered harness | | `--output table` | Render a human-readable table; default | | `--output json` | Emit the complete catalog object | ## JSON schema ```json { "models": [ { "harness": "pi", "model_id": "anthropic/claude-sonnet-4-6", "kind": "exact", "display_name": "Claude Sonnet 4.6" } ], "source": "harness-registry", "degraded": false } ``` An empty catalog keeps the same object with an empty `models` array. ## Data source The command reads harness model JSON files packaged under `observal_shared/harness_models/`. It does not contact the Observal server. An unknown harness is a usage error with exit code 2. JSON errors are written to stderr while stdout remains empty. ## Examples ```bash observal registry models observal registry models --harness pi --output json observal registry models list --harness claude-code --output json ``` # observal agent Create, compose, publish, install, and govern agents. An agent bundles MCP servers, skills, hooks, prompts, and sandboxes into one versioned Registry object. Canonical identities use `namespace/slug`. Commands also accept UUIDs, unambiguous bare names, aliases, and row numbers from the latest Agent list. ## Commands | Command | Purpose | | ---------------- | --------------------------------------------------------- | | `create` | Create from flags, JSON, or an interactive wizard | | `bulk-create` | Validate or create multiple agents from JSON | | `list` | List approved visible agents | | `my` | List agents owned or co-authored by the user | | `show` | Show an agent and its composition | | `install` | Generate an installation config without writing files | | `pull` | Generate and write a complete harness installation | | `archive` | Archive an agent | | `delete` | Compatibility alias for `archive` | | `unarchive` | Restore an archived agent | | `init` | Create a local `observal-agent.yaml` | | `add` | Add a Registry component UUID to local YAML | | `build` | Validate local composition and target scope | | `publish` | Create, update, save, or submit an agent | | `release` | Publish a reviewed version bump | | `versions` | List version history | | `outdated` | Show which pinned components have newer approved releases | | `transfer-owner` | Transfer ownership | | `co-authors` | List, add, or remove co-authors | Every leaf command supports `--output table|json`. JSON success output contains no Rich text, prompt, banner, or spinner. JSON failures leave stdout empty and write one categorized object to stderr. ## Create Use complete flags for automation: ```bash observal agent create \ --name reviewer \ --description 'Reviews changes' \ --prompt 'Review carefully.' \ --model claude-sonnet-4 \ --harness kiro \ --output json ``` Other modes: ```bash observal agent create --from-file agent.json --output json observal agent create ``` The no-flag form is interactive and cannot run in JSON mode. Flag mode requires `--name` and either `--prompt` or `--prompt-file`. Versions must be semantic versions and every repeated `--harness` must be registered. `--team HANDLE --visibility team` creates a private teamspace agent. `--visibility team` requires `--team`. JSON returns the direct server Agent object. ## Bulk create Input may be an array or an object containing an `agents` array: ```json { "agents": [ { "name": "reviewer", "version": "1.0.0", "owner": "alice", "prompt": "Review carefully.", "model_name": "claude-sonnet-4", "components": [] } ] } ``` ```bash observal agent bulk-create --from-file agents.json --dry-run --output json observal agent bulk-create --from-file agents.json --yes --output json ``` Input contains 1-50 agents. The CLI rejects malformed fields and duplicate canonical names before HTTP, and the server applies the same duplicate rules. JSON mode requires either `--dry-run` or `--yes`. Results include per-agent statuses, summary counts, and `partial`. Item errors set `partial: true` and exit with code 11; skips alone remain successful. ## List, my, and show ```bash observal agent list --search 'incident response' --output json observal agent list --namespace alice --page 2 --limit 20 --output json observal agent list --team platform --output json observal agent my --output json observal agent show alice/reviewer --output json ``` `list` JSON is paginated: ```json { "items": [], "total": 0, "page": 1, "page_size": 50 } ``` The most recent `list` or `my` result is cached for row-number references. Empty results clear stale Agent rows. `--interactive` is human-only and cannot be combined with JSON. `my` returns the standard `items`, `total`, `page`, and `page_size` envelope, including pending, approved, rejected, and archived agents. Empty results use `items: []`. `show` returns the direct Agent object, including component links and success criteria. ## Generate installation config `install` asks the server to generate config but does not write it: ```bash observal agent install alice/reviewer --harness kiro --output json ``` JSON returns the complete server installation result. Legacy `--raw` prints only `config_snippet`; new automation should use the shared output option. Use [`observal agent pull`](/cli-reference/pull) to write and track the generated installation. ## Archive, delete, and restore ```bash observal agent archive alice/reviewer --yes --output json observal agent delete alice/reviewer --yes --output json observal agent unarchive alice/reviewer --yes --output json ``` `delete` is an alias for the same reversible archive operation. JSON mode never prompts and requires `--yes`. JSON returns the direct updated Agent object. ## Local authoring workflow ### Initialize ```bash observal agent init \ --dir ./reviewer \ --name reviewer \ --description 'Reviews changes' \ --prompt-file ./PROMPT.md \ --model claude-sonnet-4 \ --harness kiro \ --output json ``` The no-flag form is interactive. JSON mode requires complete flags and refuses to overwrite an existing definition. Writes are atomic. #### From a working session ```bash observal agent init --from-capabilities --name pr-review-flow --dir ./pr-review-flow --output json observal agent init --from-capabilities --since 3d --name data-helper --output json ``` `--from-capabilities` pre-fills `components` and `supported_harnesses` from the resources recorded in the capability lock for the current directory (see [`observal discover`](/cli-reference/discover)): everything loaded with `discover use` and every component installed here in the last 24 hours (`--since` accepts `45m`, `2h`, `3d`, `1w`). In JSON mode a default description and prompt are filled in when omitted. Agents cannot nest other agents, so a pulled agent is reported under `from_capabilities.skipped_agents` rather than added. Exit code `5` when nothing was used in the window. JSON returns: ```json { "path": "/work/reviewer/observal-agent.yaml", "agent": { "name": "reviewer", "version": "1.0.0" } } ``` ### Add components Find a component in Registry JSON, then copy its UUID: ```bash observal registry skill list --search review --output json observal agent add skill 22222222-2222-2222-2222-222222222222 --dir ./reviewer --output json ``` Valid types are `mcp`, `skill`, `hook`, `prompt`, and `sandbox`. Duplicate type and UUID pairs return conflict exit code 6. Invalid types or IDs return validation exit code 7. Pin an exact component release with `--version`: ```bash observal agent add mcp 33333333-3333-3333-3333-333333333333 --version 1.4.2 --dir ./reviewer --output json ``` This writes `version: 1.4.2` on the component entry. Without a version, the Agent keeps the version its current release pins, or pins the latest approved release when it first gains the component. Versions are matched exactly; `latest` and ranges are not accepted. A release pins approved versions only; a draft may also pin a component release still in review, but only one you can read (your own component, or as an admin or reviewer). JSON returns the YAML path and added component. ### Build ```bash observal agent build --dir ./reviewer --output json observal agent build --dir ./reviewer --team platform --visibility team --output json ``` Build verifies every component, checks that any pinned component `version` exists, and validates whether private components are visible to the target owner. A successful JSON result contains `valid`, `agent`, `components`, and `issues`. Invalid composition exits with code 7 and leaves JSON stdout empty; the stderr error includes the same repairable data under `error.result`. ### Publish ```bash observal agent publish --dir ./reviewer --output json observal agent publish --dir ./reviewer --draft --output json observal agent publish --submit alice/reviewer --output json observal agent publish --dir ./reviewer --update --bump minor --output json ``` `--draft` and `--submit` are mutually exclusive. Scope changes cannot be combined with `--update`; use ownership transfer or a separate visibility operation. Non-interactive updates may use `--bump patch|minor|major`. `--update` edits the latest version in place, so it only works while that version is a draft, pending, or rejected. An approved version is immutable: installs and locks depend on it. The server answers conflict exit code 6 and points to `observal agent release`, which publishes a new version for review. JSON returns the direct created, saved, submitted, or updated Agent object. ## Release and versions ```bash observal agent release alice/reviewer --bump patch --dir ./reviewer --output json observal agent versions alice/reviewer --page 1 --page-size 50 --output json ``` Release obtains the server's semantic-version suggestion, submits the complete YAML snapshot, then updates local YAML atomically only after the server accepts the release. A failed server request leaves the local version unchanged. Every release pins each component to an exact version, and those pins are what every pull of the release installs. A release keeps the pins of the Agent's current release, so components never move unless the author asks: * give a component a `version` in the YAML to pin that release; * pass `--refresh-components` to move every other component to its latest approved release. ```bash observal agent release alice/reviewer --bump minor --dir ./reviewer --refresh-components --output json ``` Approval freezes the release's lock. The review gate checks that each pinned component version is approved. ## Check component pins ```bash observal agent outdated alice/reviewer observal agent outdated alice/reviewer --version 1.2.0 --output json ``` Lists each component the Agent version pins with its pinned version, its latest approved release, and a status: `current`, `outdated`, or `unlocked` (Agent versions released before pinning). Archived components are marked. Defaults to the latest Agent version. JSON returns `agent_id`, `qualified_name`, `version`, `components`, and a `summary` with `total`, `outdated`, `unlocked`, and `archived` counts. Consumers keep installing the pins; this is the author's cue to release a refreshed version. Versions JSON returns the direct paginated server object. Page size is 1 through 100. ## Transfer ownership ```bash observal agent transfer-owner alice/reviewer bob --yes --output json ``` The username may optionally begin with `@`. JSON mode requires `--yes` and returns the direct server result. ## Co-authors Co-authors can edit and publish the same Agent: ```bash observal agent co-authors list alice/reviewer --output json observal agent co-authors add alice/reviewer dev@example.com --output json observal agent co-authors add alice/reviewer @dev --output json observal agent co-authors remove alice/reviewer 550e8400-e29b-41d4-a716-446655440000 --output json ``` List returns the standard list envelope. Add returns the added user. Remove requires the user UUID returned by list and returns the direct deletion result. ## Pull ```bash observal agent pull alice/reviewer --harness kiro --no-prompt --output json ``` Pull writes harness files, records the exact Agent and component versions in `observal.lock` and the local lockfile, and reports every file and setup action. Later pulls keep the locked version until `--upgrade` or `--version`. See the [Pull reference](/cli-reference/pull) for pinning, strict mode, path, secret, merge, dry-run, and JSON behavior. ## Exit codes Common Agent failures use: | Code | Meaning | | ---- | ------------------------------------------------------------------------------- | | 3 | Authentication required or failed | | 4 | Permission denied | | 5 | Agent, component, or local definition not found | | 6 | Existing definition, duplicate component, or unsafe config merge conflict | | 7 | Invalid name, version, harness, component, scope, or command combination | | 8 | Server rate limit | | 9 | Server, filesystem, lockfile, generated config, or setup dependency unavailable | | 10 | CLI and server version mismatch | | 11 | Bulk operation returned one or more item errors | ## Related * [`observal agent pull`](/cli-reference/pull): install into a harness * [`observal registry`](/cli-reference/registry): manage Agent components * [`observal registry models`](/cli-reference/models): inspect exact harness model IDs # observal team Create and govern teamspaces, members, visibility reviews, join requests, and private-team invitations. Every leaf command supports `--output table|json`. JSON success output contains no Rich text or prompts. JSON failures leave stdout empty and write one categorized error object to stderr. ## Commands | Command | Purpose | | -------------------------- | ----------------------------------------------- | | `list` | List the signed-in user's teamspaces | | `show` | Show a teamspace and its members | | `create` | Create a teamspace | | `claim-personal` | Claim or return the caller's personal teamspace | | `delete` | Permanently delete a teamspace | | `leave` | Leave a teamspace | | `visibility set` | Change visibility or request public review | | `visibility list-requests` | List pending public visibility requests | | `visibility approve` | Approve pending public visibility | | `visibility reject` | Reject pending public visibility | | `request join` | Request membership | | `request list` | List a teamspace's join requests | | `request mine` | View the caller's request history and status | | `request withdraw` | Withdraw the caller's pending request | | `request approve` | Approve a pending request | | `request reject` | Reject a pending request | | `members list` | List members | | `members add` | Add a member or update a role | | `members remove` | Remove a member | | `invite create` | Create a private-team invitation | | `invite list` | List invitations | | `invite revoke` | Revoke an invitation | | `invite preview` | Preview an invitation token | | `invite request` | Use a token to request access | | `invite delete` | Delete an unused invitation | | `invite requests` | List requests associated with an invitation | Team references may be UUIDs, handles, or `@handle`. Unknown teamspaces use not-found exit code 5. ## List, show, and claim ```bash observal team list --output json observal team list --all --output json observal team show platform-tools --output json observal team claim-personal --output json ``` `list` returns the standard `items`, `total`, `page`, and `page_size` envelope. The default includes teamspaces where the user is a member. `--all` requests all teamspaces visible to the caller. Empty results use `items: []` and `page_size: 0`. `show` returns a combined result: ```json { "team": { "id": "11111111-1111-1111-1111-111111111111", "name": "Platform Tools", "handle": "platform-tools", "role": "owner" }, "members": [] } ``` `claim-personal` is idempotent. It creates or returns the caller's one private personal teamspace and returns the direct Team object. ## Create and visibility ```bash observal team create 'Platform Tools' \ --handle platform-tools \ --description 'Internal tooling' \ --visibility private \ --output json observal team visibility set platform-tools public --output json ``` Visibility is `public` or `private`. Creating a public teamspace or setting a private teamspace to public submits a review request. Until approval, the response reports `visibility: "private"` and `visibility_request_status: "pending"`. Create and visibility JSON return the direct Team object. Reviewers and deployment admins manage pending public visibility requests: ```bash observal team visibility list-requests --output json observal team visibility approve platform-tools --output json observal team visibility reject platform-tools \ --reason 'Add a public description' \ --output json ``` The list returns the standard list envelope. Approve and reject return the direct Team object. Approval makes the teamspace public and revokes its private invitation links. A rejection reason is optional and accepts up to 500 characters. ## Delete and leave ```bash observal team delete platform-tools --yes --output json observal team leave platform-tools --yes --output json ``` Delete is permanent. Leave removes only the caller's membership. The last owner cannot leave. Human mode prompts unless `--yes` is supplied. JSON mode never prompts and requires `--yes`. Both endpoints currently return an empty JSON object on success. ## Join requests Request access to a visible teamspace: ```bash observal team request join platform-tools \ --message 'I maintain deployments' \ --output json ``` The message is optional and limited to 500 characters. JSON returns the created join request. View your own status or withdraw the sole pending request: ```bash observal team request mine platform-tools --output json observal team request withdraw platform-tools --yes --output json ``` `request mine` returns the standard list envelope with requests ordered newest first. `request withdraw` finds the caller's pending request and marks it cancelled. Human mode prompts unless `--yes` is supplied. JSON mode requires `--yes` and returns an empty object. Owners and deployment admins can list and decide requests: ```bash observal team request list platform-tools --status pending --output json observal team request approve platform-tools @alice --output json observal team request reject platform-tools bob@example.com \ --reason 'Use the SRE teamspace' \ --output json ``` Valid status filters are `pending`, `approved`, `rejected`, and `cancelled`. Approve and reject select a pending request by exact email or case-insensitive username. A missing pending request uses not-found exit code 5. The list returns the standard list envelope. Join, approve, and reject return the direct join-request object. ## Members ```bash observal team members list platform-tools --output json observal team members add platform-tools alice@example.com --role reviewer --output json observal team members add platform-tools @bob --role owner --output json observal team members remove platform-tools @bob --yes --output json ``` Roles are `member`, `reviewer`, and `owner`. Adding an existing member updates the role. The last owner cannot be removed. Member list returns the standard list envelope. Add returns the saved member. Remove currently returns an empty object. Human remove prompts unless `--yes` is supplied. JSON remove requires `--yes`. ## Private-team invitations Create and list invitations: ```bash observal team invite create platform-tools \ --name onboarding \ --expires-days 30 \ --max-uses 20 \ --output json observal team invite list platform-tools --output json ``` `--expires-days` accepts 1 through 365. `--max-uses` accepts 1 through 10,000 or may be omitted for no use limit. Invite names accept 1 through 100 characters. Create returns the direct invitation object, including the one-time token and URL. Treat both as secrets. List returns the standard list envelope. States include active, expired, exhausted, and revoked. A recipient can preview the token, then submit an owner-reviewed access request: ```bash observal team invite preview INVITE_TOKEN --output json observal team invite request INVITE_TOKEN \ --message 'I am joining the deployment rotation' \ --output json ``` `invite preview` does not mutate membership or request state. `invite request` does not grant membership. It creates a pending join request that an owner must approve. Neither command prints the token in human output or errors. Owners and admins can inspect invitation usage: ```bash observal team invite requests \ platform-tools \ 550e8400-e29b-41d4-a716-446655440000 \ --output json ``` The command returns the standard list envelope for requests associated with that invitation. Empty results use `items: []`. Revoke an invitation while retaining its audit history: ```bash observal team invite revoke \ platform-tools \ 550e8400-e29b-41d4-a716-446655440000 \ --yes \ --output json ``` Delete an invitation only when it has no uses or request history: ```bash observal team invite delete \ platform-tools \ 550e8400-e29b-41d4-a716-446655440000 \ --yes \ --output json ``` The invite ID must be a UUID. Human mode prompts unless `--yes` is supplied. JSON mode requires `--yes`. Revoke returns the invitation object. Delete returns an empty object. ## Exit codes | Code | Meaning | | ---- | ---------------------------------------------------------------------------------------- | | 3 | Authentication required or failed | | 4 | Membership, owner, reviewer, or admin permission denied | | 5 | Teamspace, invitation, member, or request not found | | 6 | Handle, membership, owner, visibility, invitation, or request state conflict | | 7 | Invalid visibility, role, status, UUID, text length, token, or missing JSON confirmation | | 8 | Rate limit reached | | 9 | Server unavailable | | 10 | CLI and server version mismatch | ## Related * [`observal inbox`](/cli-reference/inbox): view request and visibility decisions * [`observal agent`](/cli-reference/agent): publish Agents to a teamspace * [`observal registry`](/cli-reference/registry): publish components to a teamspace # observal ops Inspect current session telemetry, Registry usage, feedback, logs, and Agent insight reports. All 11 supported workflows provide table and JSON output. Finite commands return one JSON document. Streaming commands return JSON Lines, one compact object per update. The dead `ops metrics`, legacy `ops spans`, and synthetic `ops telemetry test` commands have been removed. Current trace detail comes from `ops traces --span` through the Sessions API. ## Commands | Command | Purpose | | ------------------- | ------------------------------------------------- | | `top` | Show the most-downloaded MCPs or Agents | | `rate` | Submit Registry feedback | | `rate-update` | Update the caller's feedback | | `rate-delete` | Delete the caller's feedback | | `feedback` | Show aggregate and individual feedback | | `traces` | List current sessions or fetch their details | | `telemetry status` | Check server event counts and local outbox health | | `logs` | Follow local or remote development logs | | `insights list` | List Agent insight reports | | `insights show` | Show an insight report or section | | `insights generate` | Generate an insight report | ## Top Registry items ```bash observal ops top --type mcp --output json observal ops top --type agent --output json ``` JSON returns the direct ranking array. MCP ranking is currently server-limited to five results; Agent ranking is currently server-limited to six. ## Feedback Submit feedback: ```bash observal ops rate alice/postgres --type mcp --stars 5 --comment 'Reliable' --output json ``` Update feedback: ```bash observal ops rate-update alice/postgres --type mcp --stars 4 --output json observal ops rate-update alice/postgres --type mcp --comment 'Updated review' --anonymous --output json ``` Delete feedback: ```bash observal ops rate-delete alice/postgres --type mcp --yes --output json ``` View feedback: ```bash observal ops feedback alice/postgres --type mcp --output json ``` Feedback types are `mcp`, `agent`, `skill`, `hook`, `prompt`, and `sandbox`. Stars accept 1 through 5 and comments accept at most 5,000 characters. Rate and update return the direct feedback object. Delete currently returns an empty object. JSON deletion requires `--yes`; human mode prompts without it. Feedback JSON combines aggregate and individual results: ```json { "summary": { "average_rating": 4.5, "total_reviews": 2 }, "reviews": [] } ``` ## Traces and current session details `traces` uses the current REST session pipeline: * Summary list: `GET /api/v1/sessions` * Turn or span detail: `GET /api/v1/sessions/{session_id}` ```bash observal ops traces --limit 20 --output json observal ops traces --platform kiro --days 7 --output json observal ops traces --turn --limit 5 --output json observal ops traces --span --limit 3 --output json ``` `--platform` accepts a registered harness. `--days` accepts 1 through 365 and `--limit` accepts 1 through 200. Default JSON returns the direct session summary array. Turn and span JSON fetch every selected session detail: ```json { "view": "span", "items": [ { "summary": { "session_id": "session-id" }, "detail": { "events": [] } } ] } ``` `--turn` renders prompts and tool calls. `--span` includes full assistant and tool-result detail. Detail failures are surfaced rather than replaced with incomplete summaries. ## Telemetry status ```bash observal ops telemetry status --output json ``` The server portion requires administrator access. The result combines recent server counts and local durable outbox state: ```json { "server": { "status": "ok", "tool_call_events": 8, "agent_interaction_events": 5 }, "outbox": { "available": true, "pending": 0, "failed": 0, "total": 0, "bytes": 0 } } ``` A local outbox read failure remains visible as `available: false` with a safe error type. The removed synthetic test command is not part of the current JSONL session ingestion architecture. ## Logs Read a finite local tail: ```bash observal ops logs --level WARNING --lines 50 --no-follow --output json ``` Follow local logs: ```bash observal ops logs --level INFO --output json ``` Follow remote server logs with administrator access: ```bash observal ops logs --remote --level WARNING --output json ``` Valid levels are `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. `--filter` performs case-insensitive text filtering. `--lines` must be zero or greater. JSON mode emits JSON Lines. Local records have this shape: ```json {"event":"log","source":"local","level":"ERROR","line":"raw log line"} ``` Remote records wrap the structured server event: ```json {"event":"log","source":"remote","log":{"level":"ERROR","event":"request failed"}} ``` Malformed remote records, inaccessible files, HTTP errors, timeouts, and connection failures use categorized exits. Access tokens are never emitted. Log content itself may contain sensitive application data and must be handled accordingly. ## Agent insights List reports: ```bash observal ops insights list alice/reviewer --output json ``` Show a report or one section: ```bash observal ops insights show alice/reviewer latest --output json observal ops insights show alice/reviewer latest --section suggestions --output json ``` A section request returns `report_id`, `section`, and that section's `data` rather than the entire report. Generate once: ```bash observal ops insights generate alice/reviewer --period 14 --output json ``` Generate and wait with JSON Lines progress: ```bash observal ops insights generate alice/reviewer --period 14 --wait --output json ``` ```json {"event":"queued","report":{"id":"report-id","status":"pending"}} {"event":"progress","report":{"id":"report-id","status":"running","progress_percent":50}} {"event":"progress","report":{"id":"report-id","status":"completed","progress_percent":100}} ``` Periods accept 1 through 365 days. `--version` and `--compare` require semantic versions. Invalid report rows, ambiguous prefixes, missing reports, unknown sections, unavailable providers, generation failures, and wait timeouts use categorized errors. Report sections include: * `at_a_glance` * `what_they_work_on` * `interaction_style` * `usage_patterns` * `what_works` * `friction_analysis` * `suggestions` * `usage_cost_analysis` * `version_comparison` * `regression_detection` * `on_the_horizon` * `fun_ending` ## Exit codes | Code | Meaning | | ---- | --------------------------------------------------------------------------------- | | 3 | Authentication required or failed | | 4 | Telemetry, log, session, or insight permission denied | | 5 | Registry item, report, session, or local log file not found | | 6 | Ambiguous report or Registry reference | | 7 | Invalid type, harness, section, period, version, comment, or missing confirmation | | 8 | Rate limit reached | | 9 | Server, provider, stream, outbox, or local log dependency unavailable | | 10 | CLI and server version mismatch | ## Related * [`observal inbox`](/cli-reference/inbox): completed insight notifications * [`observal agent`](/cli-reference/agent): apply insight recommendations * [`observal registry`](/cli-reference/registry): inspect rated or recommended components * [Debug Agent failures](/use-cases/debug-agent-failures) # observal admin Manage server settings, users, security policy, SSO, audit data, and the submission review queue. All 24 workflows support `--output table|json`. JSON mode never prompts or emits progress banners. Destructive JSON commands require `--force` or `--yes`. Core administration requires the `admin` role. Review commands are also available to global reviewers and authorized teamspace owners or reviewers. Some role changes require `super_admin`. ## Commands | Command | Purpose | | -------------------- | -------------------------------------- | | `settings` | List dynamic server settings | | `set` | Create or update a dynamic setting | | `users` | List users | | `create-user` | Create a password-auth user | | `reset-password` | Reset or generate a user password | | `delete-user` | Permanently delete a user | | `set-role` | Change a user role | | `diagnostics` | Show database, key, and runtime health | | `trace-privacy` | Show trace-redaction policy | | `trace-privacy-set` | Change trace-redaction policy | | `cache-clear` | Clear server caches | | `saml-config` | Show redacted SAML configuration | | `saml-config-set` | Create or replace SAML configuration | | `saml-config-delete` | Delete SAML configuration | | `scim-tokens` | List SCIM token metadata | | `scim-token-create` | Create a one-time SCIM bearer token | | `scim-token-revoke` | Revoke a SCIM token | | `security-events` | Query security events | | `audit-log` | Query compliance audit events | | `audit-log-export` | Export audit events as CSV or JSON | | `review list` | List pending submissions | | `review show` | Show one submission | | `review approve` | Approve a component, Agent, or bundle | | `review reject` | Reject a component, Agent, or bundle | ## Settings ```bash observal admin settings --output json observal admin set review.require_approval true --output json ``` `settings` returns the direct settings array. Sensitive values are redacted by the server. `set` returns the server setting object and never echoes the supplied value in human output. A positional setting value may be retained by shell history. Prefer deployment secret files for sensitive settings that support external management. ## Users and roles List users: ```bash observal admin users --output json ``` Create a user and let the server generate a password: ```bash observal admin create-user alice@example.com 'Alice Smith' --role user --output json ``` Supported roles are `super_admin`, `admin`, `reviewer`, and `user`. Create with a chosen password only when shell-history exposure is acceptable: ```bash observal admin create-user alice@example.com 'Alice Smith' --password 'chosen-password' --output json ``` Reset interactively in human mode: ```bash observal admin reset-password alice@example.com ``` Generate a password without prompting: ```bash observal admin reset-password alice@example.com --generate --output json ``` JSON reset requires `--generate`. Created and generated passwords are returned once. Treat the entire result as a secret and do not paste it into logs, issues, or chat. Change a role: ```bash observal admin set-role alice@example.com reviewer --output json ``` Delete a user: ```bash observal admin delete-user alice@example.com --force --output json ``` Human deletion prompts unless `--force` or `--yes` is present. JSON deletion requires confirmation through one of those flags. ## Diagnostics and policy ```bash observal admin diagnostics --output json observal admin trace-privacy --output json observal admin trace-privacy-set true --output json observal admin cache-clear --output json ``` `diagnostics` returns the direct health object. Trace privacy responses return `trace_privacy`. Cache clear returns the number of cleared entries. ## SAML Show redacted configuration: ```bash observal admin saml-config --output json ``` Create or replace configuration: ```bash observal admin saml-config-set \ --idp-entity-id 'https://idp.example.com/entity' \ --idp-sso-url 'https://idp.example.com/sso' \ --idp-x509-cert "$(cat idp-cert.pem)" \ --active \ --output json ``` Every update requires the IdP entity ID, SSO URL, and X.509 certificate. Optional flags include `--idp-slo-url`, `--sp-entity-id`, `--jit|--no-jit`, and `--active|--inactive`. Certificate and private-key material is never returned by configuration reads. Delete configuration: ```bash observal admin saml-config-delete --force --output json ``` JSON deletion requires `--force` or `--yes`. ## SCIM tokens ```bash observal admin scim-tokens --output json observal admin scim-token-create --description 'Okta' --output json observal admin scim-token-revoke 11111111-1111-1111-1111-111111111111 --force --output json ``` List results contain metadata and token prefixes only. Creation returns the plaintext bearer token once. Treat that result as a secret. Revocation requires a complete UUID and prompts in human mode unless forced. ## Security events ```bash observal admin security-events --limit 50 --offset 0 --output json observal admin security-events --type auth.login.failure --severity critical --output json observal admin security-events --actor alice@example.com --output json ``` Severity accepts `info`, `warning`, or `critical`. Limit accepts 1 through 1,000 and offset accepts zero or greater. JSON returns the server envelope with `events` and `total`. ## Audit log Query events: ```bash observal admin audit-log --limit 100 --offset 0 --output json observal admin audit-log --actor alice@example.com --resource-type agent --output json observal admin audit-log --source cli --outcome success --output json observal admin audit-log --start-date 2026-08-01 --end-date 2026-08-31 --output json ``` Available filters are action, actor, resource type, sensitivity, outcome, source, start date, and end date. Source accepts `server` or `cli`. Limit accepts 1 through 500. Print CSV to stdout: ```bash observal admin audit-log-export ``` Write CSV atomically: ```bash observal admin audit-log-export --file audit.csv ``` Print JSON: ```bash observal admin audit-log-export --output json ``` Write JSON atomically: ```bash observal admin audit-log-export --output json --file audit.json ``` Existing files prompt in human mode. JSON mode fails with a conflict unless `--force` or `--yes` is provided. Audit exports can contain sensitive administrative data. ## Review queue List pending submissions: ```bash observal admin review list --output json observal admin review list --type mcp --output json observal admin review list --tab agents --output json observal admin review list --team-id 11111111-1111-1111-1111-111111111111 --output json ``` Component types are `mcp`, `skill`, `hook`, `prompt`, and `sandbox`. Tabs are `agents` and `components`. A component type cannot be combined with the Agents tab. The list refreshes the `review` row cache, including when empty. Row numbers can then be used by other review commands. Show a submission: ```bash observal admin review show 1 --output json ``` The JSON detail may include submitted configuration, headers, or environment-variable declarations. Handle review data as potentially sensitive. Approve: ```bash observal admin review approve 1 --output json observal admin review approve AGENT_UUID --agent --output json observal admin review approve BUNDLE_UUID --bundle --output json ``` Reject with a reason containing 1 through 5,000 characters: ```bash observal admin review reject 1 --reason 'Missing environment variable documentation' --output json observal admin review reject AGENT_UUID --agent --reason 'Unsafe prompt' --output json observal admin review reject BUNDLE_UUID --bundle --reason 'License conflict' --output json ``` `--agent` and `--bundle` are mutually exclusive. Approval and rejection return the direct server decision object. ## Exit codes | Code | Meaning | | ---- | --------------------------------------------------------------------------------------- | | 3 | Authentication required or failed | | 4 | Administrator or reviewer permission denied | | 5 | User, token, setting, review, or configuration not found | | 6 | Ambiguous reference or existing export conflict | | 7 | Invalid role, filter, UUID, SAML input, reason, or missing non-interactive confirmation | | 8 | Rate limit reached | | 9 | Server, database, ClickHouse, Redis, or filesystem unavailable | | 10 | CLI and server version mismatch | ## Related * [`observal auth`](/cli-reference/auth): inspect the active account and role * [`observal inbox`](/cli-reference/inbox): review and security notifications * [`observal ops`](/cli-reference/ops): sessions, telemetry, logs, and insights # observal doctor Diagnose local Observal state, repair managed telemetry instrumentation, remove managed instrumentation, and work with redacted support bundles. All five workflows support `--output table|json`. JSON mode never prompts or emits human banners. ## Commands | Command | Purpose | | ------------------------ | ------------------------------------------------------------------------ | | `doctor` | Diagnose configuration, Registry metadata, skills, and harness telemetry | | `doctor patch` | Install or update Observal-managed telemetry instrumentation | | `doctor cleanup` | Remove Observal-managed telemetry instrumentation | | `doctor support bundle` | Generate a redacted diagnostic archive | | `doctor support inspect` | Inspect a diagnostic archive without extracting it | ## Diagnose ```bash observal doctor --output json ``` Doctor checks: * Local authentication configuration and server health * Registry lockfile metadata against the active Registry * Managed hooks, plugins, or extensions for all registered harnesses * UUID-attributed Kiro Agent hooks * Bundled Observal skill installation JSON diagnosis exits zero when the checks ran successfully. Health is reported through `healthy`, `issues`, and `warnings`: ```json { "healthy": false, "issues": ["Cannot reach the configured server"], "warnings": ["Cursor session push hooks are not installed"], "lockfile_changes": [], "skill_missing": [], "fix_attempted": false, "patch": null } ``` Human mode retains health-check behavior: unresolved issues exit nonzero. Warnings alone remain successful. Apply fixable warnings and canonical lockfile metadata without prompting: ```bash observal doctor --yes --output json ``` Installed version pins are not changed. Only Observal-managed telemetry entries are updated. ## Patch Preview every registered harness: ```bash observal doctor patch --all-harnesses --dry-run --output json ``` Patch selected harnesses: ```bash observal doctor patch --harness claude-code --harness kiro --output json ``` Exactly one target mode is required: `--all-harnesses` or one or more `--harness` options. JSON returns one result per harness: ```json { "action": "patch", "dry_run": false, "changed": true, "targets": [ {"harness": "kiro", "changed": true} ] } ``` Patch is idempotent. It preserves unrelated hooks and configuration. Configuration writes are atomic. Invalid or unreadable harness files fail loudly rather than being replaced. For Pi, patch installs the bundled TypeScript extension at `~/.pi/agent/extensions/observal.ts` and records the CLI version it came from in an adjacent `.observal-extension.json`. If `npm:observal-pi` is registered in `~/.pi/agent/settings.json`, that registration wins: patch installs nothing locally, and removes a local copy it recognises as its own so Pi does not load the extension twice. A local file Observal did not write is never touched. MCP commands and remote URLs are never wrapped or rewritten. ## Cleanup Preview cleanup: ```bash observal doctor cleanup --dry-run --output json ``` Remove instrumentation from one harness: ```bash observal doctor cleanup --harness goose --yes --output json ``` Remove instrumentation from all registered harnesses except selected entries: ```bash observal doctor cleanup --exclude kiro --yes --output json ``` Cleanup removes only Observal-managed hooks, plugins, extensions, and legacy telemetry settings. User-owned hooks remain. Human mode confirms before writing unless `--yes` is present. JSON cleanup requires `--yes` unless it is a dry run. Unknown harnesses, conflicting selections, malformed configuration, and write failures are surfaced before success is reported. ## Support bundles ```bash observal doctor support bundle --output json observal doctor support inspect ./observal-support.tar.gz --output json ``` See [Support bundles](/cli-reference/doctor/support) for archive contents, redaction, offline behavior, and inspection limits. ## Exit codes | Code | Meaning | | ---- | ------------------------------------------------------------------------------------------ | | 3 | Authentication is required for patching | | 5 | Support bundle or requested bundle file not found | | 6 | Output archive already exists | | 7 | Invalid harness selection, malformed harness file, invalid bundle, or missing confirmation | | 9 | Server, collector, or filesystem unavailable | ## Related * [`observal scan`](/cli-reference/scan): read-only harness inventory * [`observal agent pull`](/cli-reference/pull): install a complete Agent * [Session tracking](/core-concepts/session-tracking): telemetry delivery architecture # Support bundles Generate and inspect redacted diagnostic support bundles. Both workflows support `--output table|json`. Bundle generation writes a `.tar.gz` archive as its intended side effect. JSON mode returns archive metadata and never prompts. ## Generate a bundle ```bash observal doctor support bundle --output json ``` Choose the archive path with the hard-renamed `--file/-f` option: ```bash observal doctor support bundle \ --file /tmp/observal-support.tar.gz \ --logs-since 2h \ --no-include-system \ --output json ``` The former archive-path meaning of `--output/-o` has been removed. `--output/-o` now consistently selects `table` or `json`. ### Options | Option | Purpose | | ----------------------------------------- | --------------------------------------------------------------------- | | `--file`, `-f` | Archive path; defaults to a timestamped file in the current directory | | `--logs-since` | Positive duration up to 30 days, such as `30m`, `2h`, or `1d12h` | | `--include-system`, `--no-include-system` | Include or omit local system metrics | | `--force`, `--yes` | Overwrite an existing archive and accept the size warning | | `--output`, `-o` | Human or JSON result | Existing archives prompt in human mode. JSON mode fails with a conflict unless `--force` or `--yes` is present. JSON returns: ```json { "path": "/tmp/observal-support.tar.gz", "size_bytes": 43120, "remote_status": "collected", "warnings": [], "collector_results": {}, "redaction_counts": {} } ``` ## Collection behavior Remote collectors use `POST /api/v1/support/collect` and require an administrator account. They provide: * Application and migration versions * PostgreSQL, ClickHouse, and Redis health * Allowlisted configuration * Aggregate table counts * Error fingerprints * Recent redacted log records Local collectors provide allowlisted configuration processing and optional OS, CPU, memory, disk, and container-runtime details. Remote collection is optional by design. Authentication, permission, version, network, or response failures are recorded in `remote_status` and `warnings`; local collection continues. This fallback is never silent. Individual collector failures are retained in the manifest and JSON result. If no collector produces data, the command exits with code 9 and does not create an archive. ## Privacy and integrity Support bundles contain aggregate diagnostics, not customer database rows or trace payloads. Protections include: * Configuration allowlisting before archive creation * Redaction of passwords, tokens, API keys, private keys, authorization values, URL credentials, JWTs, AWS access keys, and high-entropy strings * Redaction of collector error text before it enters the manifest * Rejection of unknown collector names and unsafe archive paths * Hashed host identity rather than a raw hostname * SHA-256 inventory entries for every archived file * Atomic archive replacement with mode `0600` The manifest records collector status, redaction counts, file sizes, and hashes. Review every bundle before sharing it because logs and deployment metadata can still be operationally sensitive. ## Size budget The uncompressed budget is 100 MB. Human mode asks before exceeding it. JSON mode requires `--force` and never prompts. ## Inspect a bundle ```bash observal doctor support inspect ./observal-support.tar.gz --output json ``` Show one regular file: ```bash observal doctor support inspect ./observal-support.tar.gz \ --show health/postgres.json \ --output json ``` JSON returns the manifest, safe regular-file inventory, schema warnings, and optional shown content: ```json { "manifest": {"bundle_schema_version": "1"}, "files": [{"path": "health/postgres.json", "size_bytes": 91}], "warnings": [], "shown": {"path": "health/postgres.json", "content": "{\"status\":\"ok\"}"} } ``` Inspection never extracts files. It rejects traversal paths, links requested through `--show`, malformed or oversized manifests, and shown files larger than 1 MB. Unsafe archive members are omitted and reported as warnings. ## Exit codes | Code | Meaning | | ---- | ---------------------------------------------------------------------------------------------------------------- | | 5 | Bundle or requested member not found | | 6 | Destination archive already exists | | 7 | Invalid duration, unsafe archive, malformed manifest, oversized display, or missing non-interactive confirmation | | 9 | No diagnostic data or archive write failure | ## Related * [`observal doctor`](/cli-reference/doctor): diagnose and repair telemetry instrumentation * [`observal ops telemetry status`](/cli-reference/ops): check telemetry flow * [Troubleshooting](/self-hosting/troubleshooting): deployment diagnosis # observal self Inspect or change the installed CLI version. All four workflows support `--output table|json`. JSON upgrade, downgrade, and rollback require `--force` and never prompt. ## Commands | Command | Purpose | | ----------- | ------------------------------------------------------------------ | | `upgrade` | Install the latest or a specified newer CLI release | | `downgrade` | List releases or install an older release | | `rollback` | Restore the standalone binary saved before the last version change | | `status` | Show current version, install method, and update availability | The destructive uninstall workflow has been removed. ## Status ```bash observal self status --output json ``` JSON returns: ```json { "current_version": "2.4.0", "install_method": "uv_tool", "path": "/home/user/.local/bin/observal", "writable": true, "managed_by": "uv", "github_available": true, "latest_version": "2.5.0", "update_available": true } ``` The command always checks GitHub. An unreachable GitHub service remains a successful local status result with `github_available: false`, `latest_version: null`, and `update_available: null`. ## Upgrade Install the latest stable release: ```bash observal self upgrade --force --output json ``` Install a specific release: ```bash observal self upgrade --version 2.5.0 --force --output json ``` Include prereleases when resolving the latest version: ```bash observal self upgrade --pre --force --output json ``` A completed JSON result contains the prior version, target version, install method, and executable path. When the current version already matches the target, status is `up_to_date` and no installation occurs. The command rejects invalid versions and targets older than the current version. System-package installations must be upgraded through their package manager. ## Downgrade List available releases: ```bash observal self downgrade --list --output json ``` JSON returns `current_version` and release `items`, each with a `current` boolean. Install an older release: ```bash observal self downgrade --version 2.4.0 --force --output json ``` `--list` and `--version` are mutually exclusive. The target must be older than the current version and at least the CLI version floor. System-package installations must use their package manager. Releases before 1.10.4 are pinned by disabling their legacy automatic-update setting. The JSON result reports this as `automatic_updates_disabled`. ## Rollback ```bash observal self rollback --force --output json ``` Rollback is available only for standalone binary installations with `~/.observal/bin/observal.prev`. It acquires the same version-change lock as upgrade and downgrade, copies the backup to a temporary file, restores executable permissions, and atomically replaces the current binary. Package-managed installations must install their previous version through the package manager. ## Installation safety Upgrade and downgrade: 1. Acquire a process lock so version changes cannot overlap. 2. Use the detected installation method. 3. Verify standalone binary downloads against the published SHA-256 checksum. 4. Replace standalone binaries atomically and retain the previous binary for rollback. 5. Execute the installed binary and verify that it reports the requested version. Human mode may explicitly confirm an unsigned binary. JSON mode never prompts and refuses standalone installation when the release has no published checksum. ## Exit codes | Code | Meaning | | ---- | ------------------------------------------------------------------------------- | | 5 | Rollback backup not found | | 6 | Package-managed install or another version change holds the lock | | 7 | Invalid version, direction, conflicting mode, or missing `--force` in JSON mode | | 9 | GitHub, download, installer, checksum, verification, or filesystem failure | ## Related * [`observal server`](/cli-reference/server): manage server stack versions separately * [Upgrades](/self-hosting/upgrades): server and CLI upgrade procedures * [Environment variables](/reference/environment-variables): update-check settings # observal registry prompt Manage reusable prompt templates in the Registry. Prompts can be submitted, listed, rendered, edited, archived, restored, transferred, and shared with co-authors. Standalone prompt installation has been removed; attach prompts to agents instead. ## Commands | Command | Description | | ---------------- | ----------------------------------------- | | `submit` | Submit a prompt or save a draft | | `list` | List approved visible prompts | | `my` | List your prompts across all statuses | | `show` | Show one prompt and its template | | `render` | Render a prompt with variables | | `edit` | Edit a draft, pending, or rejected prompt | | `archive` | Archive an approved prompt | | `unarchive` | Restore an archived prompt | | `transfer-owner` | Transfer ownership | | `co-authors` | List, add, or remove co-authors | All structured commands support table and JSON output. ## Submit ```bash observal registry prompt submit \ --name review \ --description "Review code" \ --category code-review \ --template "Review {{code}}" \ --output json observal registry prompt submit --from-file prompt.json --output json observal registry prompt submit --submit acme/review --output json ``` JSON mode never prompts. A plain template file requires explicit name, description, and category options. A JSON file may contain the complete payload. Valid categories are `system-prompt`, `code-review`, `code-generation`, `testing`, `documentation`, `debugging`, and `general`. ## List and show ```bash observal registry prompt list --category code-review --output json observal registry prompt my --output json observal registry prompt show acme/review --output json ``` Row numbers are scoped to the latest Prompt list. Empty Prompt lists clear previous Prompt row references. Human output escapes template text so bracketed content is rendered literally. ## Render ```bash observal registry prompt render acme/review --var code=main.py --output json ``` Every variable must use `key=value` syntax with a non-empty key. JSON returns the direct server result. Human output prints the rendered prompt as literal text. ## Edit ```bash observal registry prompt edit acme/review --description "Updated" --output json observal registry prompt edit acme/review --from-file updates.json --output json ``` Invalid categories, versions, files, and edit-lock conflicts preserve their shared error category and stable exit code. ## Agent usage Prompts are agent components rather than standalone harness installations: ```bash observal agent add prompt observal agent build ``` ## Related * [`observal registry`](/cli-reference/registry): complete Registry reference * [`observal agent`](/cli-reference/agent): compose and publish agents # observal server Manage local Observal deployments. The lifecycle commands operate the embedded PostgreSQL, ClickHouse, Redis, and API processes. Upgrade, rollback, and version commands operate a local Docker Compose deployment. Local filesystem, process, Docker, and database access are the authorization boundary. These commands do not require a reachable Observal API or an API role. ## Command summary | Command | Target | Purpose | | ---------- | --------- | ------------------------------------------------------ | | `start` | Embedded | Start dependencies and the API | | `stop` | Embedded | Stop all services | | `restart` | Embedded | Stop and start services | | `status` | Embedded | Report service health and ports | | `logs` | Embedded | Read or follow service logs | | `install` | Embedded | Install verified dependency binaries | | `config` | Embedded | Show local paths and ports | | `reset` | Embedded | Delete database data and generated secrets | | `upgrade` | Docker | Back up PostgreSQL and replace images | | `rollback` | Docker | Restore PostgreSQL and the prior image version | | `versions` | Docker | List image versions and managed backups | | `migrate` | Databases | Move PostgreSQL registry and ClickHouse telemetry data | ## Embedded lifecycle Start in the foreground: ```bash observal server start ``` Start for automation: ```bash observal server start --background --output json ``` JSON start and restart require `--background` because foreground mode remains attached until shutdown. ```bash observal server restart --background --output json observal server stop --output json ``` `start` accepts `--port/-p` and `--host`. When the default API port is occupied, it tries the documented local fallback ports and reports the selected port. An explicitly selected occupied port is a conflict. Background start and restart verify all four final service states before returning success; stop verifies that none remains running. Their JSON results include the verified `services` list, and an unverified transition exits with code 9 plus observed state in `error.result`. Startup performs these steps in order: 1. Installs embedded dependencies when missing. Downloads require a published SHA-256 checksum and archives reject links and path traversal. 2. Starts PostgreSQL, ClickHouse, and Redis. 3. Applies PostgreSQL and ClickHouse migrations. Migration failures stop startup; the CLI never stamps a failed PostgreSQL schema as current. 4. Starts the API. 5. Bootstraps a local admin only on a fresh embedded server and persists the real access and refresh tokens. It never writes placeholder credentials or an API key. 6. Attempts telemetry hook installation. Optional hook failures are explicit warnings. ## Status and configuration ```bash observal server status --output json observal server config --output json ``` Status is a finite diagnosis command. It exits successfully when checks run, including when `healthy` is false. ```json { "healthy": false, "services": [ {"service": "postgres", "status": "running", "port": 5480}, {"service": "clickhouse", "status": "stopped", "port": 8124}, {"service": "redis", "status": "running", "port": 6380}, {"service": "api", "status": "stopped", "port": 8000} ] } ``` Configuration output contains paths and ports only. It never returns generated secrets. ## Logs Read a bounded snapshot: ```bash observal server logs --output json observal server logs api --lines 200 --output json ``` Follow one service as JSON Lines: ```bash observal server logs api --follow --output json ``` JSON follow requires one service so every event has an unambiguous `service` field. Valid services are `postgres`, `clickhouse`, `redis`, and `api`. ## Install and reset ```bash observal server install --output json observal server install --upgrade --output json ``` Reset deletes embedded database directories and the generated server secret. It does not delete CLI configuration, downloaded binaries, logs, or unrelated files. Human mode confirms. JSON mode requires `--force`: ```bash observal server reset --force --output json ``` Deletion is confined to the managed embedded data directory. ## Docker upgrade Preview an upgrade: ```bash observal server upgrade --dry-run --output json ``` Apply one non-interactively: ```bash observal server upgrade --version 1.2.3 --force --output json ``` An upgrade validates the target version and image, acquires the server upgrade lock, creates a managed PostgreSQL backup unless `--skip-backup` is set, pulls images, atomically updates `OBSERVAL_VERSION`, recreates containers, and runs the configured health check. A failed health check requests the previous image version again and returns an unavailable error. JSON mutation requires `--force`; dry run does not. ## Docker rollback ```bash observal server rollback --force --output json observal server rollback \ --from-backup ~/.observal/backups/v1.2.2-20260521T120000 \ --force --output json ``` Rollback accepts only backup directories under the managed backup root. It restores PostgreSQL, atomically restores the image version, recreates containers, and checks health. **ClickHouse telemetry is not restored by this command.** JSON and human results state `clickhouse_restored: false`. Use [`observal server migrate`](/cli-reference/server/migrate) for ClickHouse export and import. ## Docker versions ```bash observal server versions --output json ``` The result distinguishes the current version, available GHCR images, and local PostgreSQL backups. Failure to query GHCR is reported as unavailable rather than as an empty registry. ## Error and output contract All finite commands accept `--output table|json`. JSON success writes one document to stdout. Failures leave stdout empty and write one categorized error to stderr. Log following is the only JSON Lines stream. | Code | Category | Typical server cause | | ---- | ----------- | --------------------------------------------------------- | | 2 | Usage | Invalid option or missing required argument | | 4 | Permission | Unreadable local state or backup outside the managed root | | 5 | Not found | Missing logs, Compose deployment, image, or backup | | 6 | Conflict | Busy port, active upgrade lock, or existing destination | | 7 | Validation | Missing JSON confirmation or invalid version | | 9 | Unavailable | Dependency, Docker, migration, health, or network failure | ## Related * [Database migration](/cli-reference/server/migrate) * [Self-hosted upgrades](/self-hosting/upgrades) * [Backup and restore](/self-hosting/backup-and-restore) * [`observal self`](/cli-reference/self), for CLI binary versions # observal server migrate Move PostgreSQL registry data and ClickHouse telemetry between Observal deployments. Migration uses the supplied database connections directly. Local shell and database access are the authorization boundary; the command does not authenticate against a configured Observal API. Install the optional dependency first: ```bash pip install 'observal-cli[migrate]' ``` Keep connection URLs in environment variables or secret files managed by the shell. Source commands read `DATABASE_URL` and `CLICKHOUSE_URL`; target commands read `TARGET_DATABASE_URL` and `TARGET_CLICKHOUSE_URL`. Explicit URL options remain available when no secret is embedded. Do not paste credentials into shared shell history, logs, or issue reports. JSON results and categorized errors never echo a connection URL. ## Workflow 1. Export PostgreSQL. This creates a checksummed registry archive and a migration manifest. 2. Validate and import PostgreSQL on the target. 3. Export ClickHouse using the PostgreSQL migration manifest. 4. Validate and import ClickHouse on the target. PostgreSQL must be imported first so referenced users and agents exist before telemetry validation. ## PostgreSQL export ```bash observal server migrate export \ --file registry.tar.gz \ --output json ``` `--file/-f` selects the archive destination. `--output/-o` always selects `table` or `json`. Existing destinations fail with a conflict instead of being overwritten. The archive and sidecar manifest are written atomically with owner-only permissions. A partial archive is not published after failure. Example JSON fields: ```json { "archive": "registry.tar.gz", "manifest": "registry.manifest.json", "migration_id": "7b84e503-63af-4b89-a1cd-abf48f0452f3", "table_counts": {"users": 8, "agents": 21}, "total_rows": 342, "size_bytes": 1048576, "duration_seconds": 2.4 } ``` ## PostgreSQL validation and import ```bash observal server migrate validate \ --archive registry.tar.gz \ --output json observal server migrate import \ --archive registry.tar.gz \ --output json ``` Validation checks archive structure and SHA-256 checksums. When a target URL is provided, it also compares table row counts. Checksum failure returns a categorized validation error. Row-count differences remain explicit result data. Import verifies checksums before insertion. Existing rows are skipped according to the migration service's idempotent import rules. The result contains per-table inserted and skipped counts plus warnings. ## ClickHouse export ClickHouse export requires the PostgreSQL sidecar manifest and a new destination directory: ```bash observal server migrate export-telemetry \ --manifest registry.manifest.json \ --output-dir telemetry-export \ --output json ``` The destination must not already exist. This lets the exporter remove the complete directory after failure without touching pre-existing files. The directory and streamed Parquet files use restrictive permissions and atomic temporary files. The export covers active session, checkpoint, layer, audit, security, and webhook tables. Older sources may omit tables. Telemetry is divided into deterministic time-and-hash chunks so each ClickHouse query and Parquet file stays bounded independently of the total dataset size. Oversized or memory-limited chunks are split again automatically. `telemetry_manifest.json` records every chunk's range, shard, checksum, size, row count, and migration ID. ## ClickHouse validation and import ```bash observal server migrate validate-telemetry \ --input-dir telemetry-export \ --output json observal server migrate import-telemetry \ --input-dir telemetry-export \ --output json ``` Telemetry validation checks: * Parquet checksums * Manifest row counts against target ClickHouse when supplied * Agent and user references against target PostgreSQL when supplied Checksum failure is fatal. Row-count differences and orphan groups are returned explicitly. Telemetry import is resumable per Parquet chunk. Successfully imported chunk IDs and checksums are recorded in `.import_state.json` in the input directory, so a retry skips only verified completed chunks rather than an entire table or month. Deterministic insert tokens protect ambiguous retries on current non-replicated MergeTree schemas. Imported project-keyed rows normalize to the deployment project `default`, and session summaries are rebuilt after event import so summaries reflect complete sessions rather than individual insert blocks. ## Human and JSON behavior All six leaves accept `--output table|json`. Human mode renders progress and summaries. JSON mode is finite, prompt-free, suppresses progress and warnings from stdout, and returns one result document. Failures leave stdout empty and emit one categorized error to stderr. Cleartext ClickHouse transport with credentials produces a human warning. JSON mode does not print a banner; operators should use `clickhouses://` for TLS. ## Exit codes | Code | Category | Typical migration cause | | ---- | ----------- | -------------------------------------------------------------------- | | 2 | Usage | Missing required option or unsupported output mode | | 4 | Permission | Destination or source path is not accessible | | 5 | Not found | Archive, manifest, or input directory is missing | | 6 | Conflict | Archive or telemetry destination already exists | | 7 | Validation | Invalid archive, failed checksum, or missing phase prerequisite | | 9 | Unavailable | Optional dependency, database, network, or migration service failure | ## Recommended sequence ```bash # Source observal server migrate export --file registry.tar.gz --output json observal server migrate export-telemetry \ --manifest registry.manifest.json \ --output-dir telemetry-export \ --output json # Target observal server migrate validate --archive registry.tar.gz --output json observal server migrate import --archive registry.tar.gz --output json observal server migrate validate-telemetry \ --input-dir telemetry-export \ --output json observal server migrate import-telemetry \ --input-dir telemetry-export \ --output json ``` # observal registry skill Submit, browse, install, edit, archive, restore, transfer, and manage co-authors for portable skill packages. ## Commands | Command | Description | | ---------------- | -------------------------------------------- | | `submit` | Submit a Git-backed or registry-direct skill | | `list` | List approved visible skills | | `my` | List your skills across all statuses | | `show` | Show one skill | | `install` | Write a skill into a supported harness | | `edit` | Edit a draft, pending, or rejected skill | | `archive` | Archive an approved skill | | `unarchive` | Restore an archived skill | | `transfer-owner` | Transfer ownership | | `co-authors` | List, add, or remove co-authors | Every command that returns structured data supports table and JSON output. Archive, restore, and ownership transfer require the confirmation bypass in JSON mode. ## Submit Git-backed skills require a Git URL. Registry-direct skills store SKILL.md and an optional script in Observal. ```bash observal registry skill submit \ --skill-md ./SKILL.md \ --git-url https://github.com/acme/review-skill \ --name review-skill \ --description "Review code" \ --task-type code-review \ --output json observal registry skill submit \ --skill-md ./SKILL.md \ --delivery-mode registry_direct \ --name review-skill \ --description "Review code" \ --task-type code-review \ --output json ``` JSON mode never prompts. Supply the name, description, task type, delivery mode, and required source inputs explicitly. Valid task types are `code-review`, `code-generation`, `testing`, `documentation`, `debugging`, `refactoring`, `deployment`, `security-audit`, `performance`, and `general`. ## List and show ```bash observal registry skill list --task-type code-review --output json observal registry skill list --harness claude-code --output json observal registry skill my --output json observal registry skill show acme/review-skill --output json ``` Row numbers are scoped to the latest Skill list. Empty lists clear previous Skill row references. ## Install ```bash observal registry skill install acme/review-skill --harness claude-code --scope user --output json observal registry skill install acme/review-skill --harness pi --scope project --output json observal registry skill install acme/review-skill --harness pi --no-write --output json observal registry skill install acme/review-skill --harness pi --raw ``` JSON output performs the installation unless no-write is selected. It reports `write_performed` and `installed_path`. Raw mode emits only the generated config and performs no write. Lockfile state is recorded only after the skill content is written successfully. The command fails if the harness lacks skill support, the source cannot be installed, a project symlink cannot be created, or installed state cannot be recorded. ## Edit ```bash observal registry skill edit acme/review-skill --description "Updated" --output json observal registry skill edit acme/review-skill --from-file updates.json --output json ``` Edit-lock conflicts preserve conflict exit code 6. Invalid fields and files use the shared validation, not-found, permission, and unavailable categories. ## Related * [`observal registry`](/cli-reference/registry): complete registry reference * [`observal agent`](/cli-reference/agent): attach skills to agents # Overview Run Observal entirely on your own infrastructure. No SaaS, no egress, every byte of telemetry stays inside your network. For a packaged single-node install, `install-server.sh` generates restricted credential files under `secrets/`, records their `NAME_FILE` paths in `.env`, and binds host ports to loopback. It prompts when a terminal is available and applies the same safe defaults automatically in headless CI or agent environments. The source Compose path remains available for development and accepts direct `.env` values. ## Architecture at a glance ```mermaid flowchart TB engineers[Engineers] web["observal-web - static web UI"] api["observal-api - FastAPI backend"] worker["observal-worker - async jobs"] db["observal-db - Postgres"] redis["observal-redis - jobs + pub/sub"] ch["observal-clickhouse - telemetry"] grafana["observal-grafana - optional dashboards"] engineers --> web web <--> api api --> worker api --> db api --> redis api --> ch worker --> db worker --> redis worker --> ch grafana --> ch ``` **Seven services:** | Service | Image | Ports | Purpose | | --------------------- | ----------------------------------- | ---------- | ------------------------- | | `observal-api` | built from `docker/Dockerfile.api` | 8000 | FastAPI backend | | `observal-web` | built from `docker/Dockerfile.web` | 3000 | Next.js web UI | | `observal-db` | `postgres:16` | 5432 | Registry, users, config | | `observal-clickhouse` | `clickhouse/clickhouse-server:26.3` | 8123 | Session and audit events | | `observal-redis` | `redis:7-alpine` | 6379 | Job queue (arq) + pub/sub | | `observal-worker` | built from `docker/Dockerfile.api` | (internal) | Background async jobs | | `observal-grafana` | `grafana/grafana-oss:11.6.5` | 3001 | Dashboards (optional) | All services run on a private `observal-net` bridge network. Named volumes (`pgdata`, `chdata`, `redisdata`, `grafanadata`, `apidata`) hold persistent data. ## Deployment tiers Choose the deployment model that fits your team: | | [Single-node](/self-hosting/single-node-deploy) | [Kubernetes](https://github.com/Observal/Observal/blob/main/docs/self-hosting/kubernetes-helm.md) | [Production](/self-hosting/production-deploy) | | ------------------ | ----------------------------------------------- | ------------------------------------------------------------------------------------------------- | --------------------------------------------- | | **How** | Docker Compose on one VM | Official Helm chart | Terraform on AWS or GCP | | **Best for** | ≤50 users, internal tools, POCs | Cloud-native teams, existing K8s infra | Enterprise, SLA-bound, 50+ users | | **Cost** | $20 to $150/mo | Variable | \~$180 to $255/mo | | **HA** | No | Pod resilience & horizontal scaling | Yes (Multi-AZ databases, autoscaling) | | **Time to deploy** | 10 minutes | 10 to 15 minutes | 20 to 30 minutes | **Start here:** | If you want to... | Read | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Deploy on a single VM (simplest) | [Single-node deployment](/self-hosting/single-node-deploy) | | Deploy on Kubernetes with Helm | [Kubernetes deployment with Helm](https://github.com/Observal/Observal/blob/main/docs/self-hosting/kubernetes-helm.md) | | Deploy a production HA stack | [Production deployment](/self-hosting/production-deploy) | | Deploy on AWS specifically | [AWS deployment with Terraform](/self-hosting/aws-terraform) | | Deploy on GCP specifically | [GCP deployment with Terraform](/self-hosting/gcp-terraform) | **Then configure and operate:** | If you want to... | Read | | ------------------------------------- | ---------------------------------------------------------------------- | | Confirm your machine can run Observal | [Requirements](/self-hosting/requirements) | | Get the stack running locally for dev | [Docker Compose setup](/self-hosting/docker-compose) | | Know every env var that matters | [Configuration](/self-hosting/configuration) | | See every port and volume at a glance | [Ports and volumes](/self-hosting/ports-and-volumes) | | Understand the DBs and retention | [Databases](/self-hosting/databases) | | Set up SSO, JWT keys, demo accounts | [Authentication and SSO](/self-hosting/authentication) | | Understand session delivery | [Session tracking and reconciliation](/core-concepts/session-tracking) | | Upgrade safely | [Upgrades](/self-hosting/upgrades) | | Back up and restore | [Backup and restore](/self-hosting/backup-and-restore) | | Fix something that's broken | [Troubleshooting](/self-hosting/troubleshooting) | ## Production checklist Before putting Observal in front of real users: 1. **Protect `SECRET_KEY`**: server-package setup generates it in `secrets/`; source deployments must set a strong direct value or `SECRET_KEY_FILE`. 2. **Protect database passwords**: server-package setup generates restricted PostgreSQL and ClickHouse credential files; source deployments must replace the `.env.example` defaults. 3. **Scope `CORS_ALLOWED_ORIGINS`** to your real frontend host. 4. **Configure SSO** in **Admin → SSO**, including `deployment.sso_only` if you want SSO-only login. 5. **Tune rate limits** (`RATE_LIMIT_AUTH`, `RATE_LIMIT_AUTH_STRICT`). 6. **Set `DATA_RETENTION_DAYS`** to match your retention policy (default 90 days). 7. **Back up the JWT key volume** (`apidata`): losing it invalidates every session. 8. **Remove demo accounts**: remove `DEMO_*` variables and generated demo password files before a real deployment, then delete any already-seeded demo users. Each of these is covered in the linked deep-dive below. Start with [Requirements](/self-hosting/requirements). # Requirements Minimum and recommended specs for running the Observal stack. ## Hardware | Profile | CPU | Memory | Disk | Who it's for | | ---------------- | ------- | ------ | ----------------- | -------------------------------------------------------- | | Developer | 2 vCPU | 4 GB | 20 GB | One engineer, local machine | | Small team (≤10) | 2 vCPU | 6 GB | 50 GB | Small deployment, moderate telemetry | | Team (10–50) | 4 vCPU | 12 GB | 200 GB + fast SSD | Typical production deployment | | Large team (50+) | 8+ vCPU | 32 GB | 500 GB + fast SSD | High telemetry volume; consider externalizing ClickHouse | The stack's Docker memory limits out-of-the-box: | Service | Limit | | --------------------- | ------ | | `observal-api` | 512 MB | | `observal-worker` | 512 MB | | `observal-web` | 256 MB | | `observal-clickhouse` | 1 GB | | `observal-redis` | 256 MB | | `observal-grafana` | 512 MB | ClickHouse is the memory-hungry one. On a long-running team server, bump it to 2–4 GB in `docker/docker-compose.yml`. ## Disk: where the data goes The heaviest user of disk is **ClickHouse**. Growth depends on: * Number and length of harness sessions * Raw transcript record size * `DATA_RETENTION_DAYS` Session transcripts vary significantly by harness and tool output size. Measure representative workloads and plan 2 to 3 times headroom over observed growth. Postgres stays under 500 MB for most deployments; it holds only registry metadata and user accounts. ## Software | Software | Version | Notes | | ------------------ | ---------- | -------------------------------------------------------- | | Docker | ≥ 24.0 | With Compose v2 (`docker compose`, not `docker-compose`) | | Linux / macOS host | any modern | Windows via WSL2 works | | Bash / zsh | any | For the CLI install | > \[!NOTE] Homebrew's Docker formula is outdated and may ship an older Compose version. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages to get Docker Engine ≥ 24.0 with Compose v2. For the **CLI** (developer machines, not the server): * **Standalone binary** (recommended) -- no dependencies, just `curl | bash` * Or Python **3.11, 3.12, or 3.13** with `uv`, `pipx`, or `pip` ## Network * **Outbound HTTPS**: required for image downloads and enabled integrations such as OAuth, webhooks, external Git repositories, or model providers. * **Inbound**: the server package routes the UI, API, and session telemetry through nginx on port `8000`. * **Between services**: the private `observal-net` bridge handles application traffic. * **Host bindings**: new server-package installs bind nginx, the direct web port, PostgreSQL, ClickHouse, Redis, Prometheus, and Grafana to `127.0.0.1` by default. * **Secret access**: server-package setup records the operator's group as `OBSERVAL_SECRET_GID`; only that group is added to containers that need their service-specific files. ## TLS / HTTPS Local loopback access can use HTTP. For every remote deployment, terminate TLS 1.2 or later with Caddy, nginx, an enterprise load balancer, or a cloud ingress. Point the proxy at the loopback nginx listener on `localhost:8000`. Example Caddy configuration: ```caddyfile observal.your-company.internal { reverse_proxy localhost:8000 } ``` Set `OBSERVAL_BIND_ADDRESS` to a non-loopback address only when an external TLS proxy cannot reach loopback, or when plaintext access is an intentional private-network risk. Existing server-package installations retain their previous bind address during setup upgrades. See the [security assurance case](/security/assurance-case) for trust boundaries and residual network risks. ## Next → [Docker Compose setup](/self-hosting/docker-compose) # Single-node deployment Run the full Observal stack on a single VM with Docker Compose. Best for teams of up to \~50 users, internal tools, evaluations, and any deployment where simplicity matters more than multi-AZ redundancy. **End state:** Observal running behind TLS on one server, with automated backups to S3-compatible storage, surviving reboots, and upgradable in under a minute. ## When to use this vs. the Terraform module | | Single-node (this guide) | [Production Terraform](/self-hosting/aws-terraform) | | -------------------------- | ----------------------------------- | --------------------------------------------------- | | **Best for** | Small/mid teams, internal use, POCs | Enterprise, SLA-bound, high-traffic | | **Infra** | 1 VM, any cloud or on-prem | \~100 managed AWS resources | | **Cost** | $20–150/mo | \~$255/mo | | **HA** | No — single point of failure | Yes — Multi-AZ Postgres, autoscaling ECS | | **Time to deploy** | 10 minutes | 20–30 minutes | | **Operational complexity** | Low — SSH, docker compose, cron | Medium — Terraform, AWS console, CloudWatch | | **Scaling** | Vertical (bigger VM) | Horizontal (more Fargate tasks) | ## Architecture ```mermaid flowchart TB subgraph vm[Single VM] nginx["nginx - TLS"] web[Web UI] api["API - FastAPI"] worker["Worker - arq"] pg[(Postgres)] redis[(Redis)] ch[(ClickHouse)] grafana[Grafana] systemd["systemd - docker compose restart"] cron["cron - daily backups to S3"] end nginx --> web nginx --> api api --> pg api --> ch api --> redis api --> worker worker --> pg worker --> redis worker --> ch grafana --> ch systemd -.-> nginx systemd -.-> api systemd -.-> worker cron -.-> pg cron -.-> ch ``` Everything runs as Docker containers on a single host. The nginx LB routes traffic and terminates TLS. Docker's restart policy and systemd keep the stack running across reboots. ## Prerequisites | Requirement | Minimum | Recommended | | -------------------- | --------------------------------------------- | ---------------------------- | | **VM** | 2 vCPU, 4 GB RAM, 40 GB SSD | 4 vCPU, 8 GB RAM, 100 GB SSD | | **OS** | Ubuntu 22.04+ / Amazon Linux 2023 / Debian 12 | Ubuntu 24.04 LTS | | **Docker** | Engine ≥ 24.0 with Compose v2 | Latest stable | | **Domain** (for TLS) | A DNS record pointing to the VM's public IP | — | | **Firewall** | Ports 80, 443 open inbound | — | ### Sizing guide | Team size | VM spec | Estimated cost (AWS) | | ----------- | ------------------------------------------------------------ | -------------------- | | 1–10 users | `t3.medium` (2 vCPU / 4 GB) | \~$30/mo | | 10–30 users | `t3.large` (2 vCPU / 8 GB) | \~$60/mo | | 30–50 users | `t3.xlarge` (4 vCPU / 16 GB) | \~$120/mo | | 50+ users | Consider the [Terraform module](/self-hosting/aws-terraform) | \~$255/mo | ClickHouse is the memory consumer. If you run out, increase `CLICKHOUSE_MEMORY_LIMIT` before resizing the VM. ## Step 1: Provision the VM Use any cloud provider or on-prem hypervisor. Example for AWS: ```bash aws ec2 run-instances \ --image-id ami-0c7217cdde317cfec \ --instance-type t3.large \ --key-name your-key \ --security-group-ids sg-xxxxx \ --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":100,"VolumeType":"gp3"}}]' \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=observal}]' ``` For other clouds: * **GCP:** `e2-standard-2` with 100 GB balanced persistent disk * **Azure:** `Standard_B2ms` with 100 GB Premium SSD * **Hetzner:** `CPX31` (4 vCPU / 8 GB, \~€15/mo) * **On-prem:** any Linux box meeting the specs above ## Step 2: Install Docker SSH into the VM and install Docker: ```bash ssh ubuntu@your-server-ip # Ubuntu / Debian curl -fsSL https://get.docker.com | sh sudo usermod -aG docker $USER newgrp docker # Verify docker version docker compose version ``` ## Step 3: Clone and configure ```bash git clone https://github.com/Observal/Observal.git cd Observal cp .env.example .env ``` Edit `.env` for production. At minimum, change these: ```bash # Generate a secure secret key SECRET_KEY=$(python3 -c "import secrets; print(secrets.token_urlsafe(32))") # Set strong database passwords POSTGRES_PASSWORD=$(openssl rand -base64 24) CLICKHOUSE_PASSWORD=$(openssl rand -base64 24) # Set your domain (used for CORS and OAuth redirects) CORS_ALLOWED_ORIGINS=https://observal.yourcompany.com FRONTEND_URL=https://observal.yourcompany.com # Remove demo accounts for production # Comment out or delete all DEMO_* variables ``` Write them into `.env`: ```bash sed -i "s|^SECRET_KEY=.*|SECRET_KEY=$SECRET_KEY|" .env sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$POSTGRES_PASSWORD|" .env sed -i "s|^CLICKHOUSE_PASSWORD=.*|CLICKHOUSE_PASSWORD=$CLICKHOUSE_PASSWORD|" .env ``` > SAML SSO, audit logs, and executive dashboards are included in the open-source distribution. See [Configuration](/self-hosting/configuration). For mounted credentials and private keys, use the documented [`NAME_FILE` inputs](/self-hosting/configuration#secret-files) instead of placing secret contents in `.env`. ## Step 4: Set up TLS This guide uses the source Compose stack. Every `localhost:80` example below refers to that stack's nginx load balancer. Do not expose its HTTP listener directly to remote clients. For downloadable server-package bindings, see [Requirements](/self-hosting/requirements#tls--https). ### Option A: Caddy (simplest — automatic HTTPS) ```bash sudo apt install -y caddy ``` Create `/etc/caddy/Caddyfile`: ```caddyfile observal.yourcompany.com { reverse_proxy localhost:80 } ``` ```bash sudo systemctl enable --now caddy ``` Caddy handles Let's Encrypt certificates automatically. No renewal cron needed. ### Option B: Certbot + nginx production config ```bash sudo apt install -y certbot sudo certbot certonly --standalone -d observal.yourcompany.com ``` Then use the production compose overlay which configures nginx for TLS: ```bash cd docker docker compose -f docker-compose.yml -f docker-compose.production.yml up -d ``` Set up auto-renewal: ```bash echo "0 3 * * * certbot renew --quiet && docker compose -f /home/ubuntu/Observal/docker/docker-compose.yml -f /home/ubuntu/Observal/docker/docker-compose.production.yml restart observal-lb" | sudo tee -a /etc/crontab ``` ### Option C: Behind a cloud load balancer If your VM sits behind an AWS ALB, GCP HTTPS LB, or Cloudflare, terminate TLS there and proxy to port 80 on the VM. No TLS config on the VM itself. ## Step 5: Start the stack ```bash cd ~/Observal docker compose -f docker/docker-compose.yml up -d --build ``` First build takes 3–5 minutes (pulling images, building the API and web containers). Watch the logs: ```bash docker compose -f docker/docker-compose.yml logs -f observal-init observal-api ``` Wait for: ``` observal-init | INFO - Database up to date. observal-api | INFO - Application startup complete. ``` Verify: ```bash curl -fsSL http://localhost/health # {"status":"ok","initialized":true} ``` ## Step 6: Survive reboots Docker's `restart: unless-stopped` policy (already set in `docker-compose.yml`) handles container restarts. To ensure Docker itself starts on boot: ```bash sudo systemctl enable docker ``` Test it: ```bash sudo reboot # After reconnecting: docker compose -f ~/Observal/docker/docker-compose.yml ps # All services should be running ``` ## Step 7: Set up automated backups Create a backup script: ```bash sudo mkdir -p /opt/observal-backups cat << 'SCRIPT' | sudo tee /opt/observal-backups/backup.sh #!/bin/bash set -euo pipefail BACKUP_DIR="/opt/observal-backups" DATE=$(date +%Y%m%d-%H%M) COMPOSE="docker compose -f /home/ubuntu/Observal/docker/docker-compose.yml" # Postgres $COMPOSE exec -T observal-db pg_dump -U postgres observal | gzip > "$BACKUP_DIR/pg-$DATE.sql.gz" # JWT keys $COMPOSE exec -T observal-api tar czf - -C /data keys > "$BACKUP_DIR/keys-$DATE.tar.gz" # Prune backups older than 30 days find "$BACKUP_DIR" -name "pg-*.sql.gz" -mtime +30 -delete find "$BACKUP_DIR" -name "keys-*.tar.gz" -mtime +30 -delete # Upload to S3 (optional — install awscli first) # aws s3 sync "$BACKUP_DIR" s3://your-backup-bucket/observal/ --exclude "backup.sh" echo "Backup completed: $DATE" SCRIPT sudo chmod +x /opt/observal-backups/backup.sh ``` Schedule it: ```bash echo "0 3 * * * root /opt/observal-backups/backup.sh >> /var/log/observal-backup.log 2>&1" | sudo tee /etc/cron.d/observal-backup ``` For ClickHouse (weekly, since it's larger): ```bash echo "0 4 * * 0 root docker compose -f /home/ubuntu/Observal/docker/docker-compose.yml exec -T observal-clickhouse clickhouse-client --password \$CLICKHOUSE_PASSWORD --query \"BACKUP DATABASE observal TO Disk('backups', 'weekly-\$(date +\%Y\%m\%d).zip')\" >> /var/log/observal-backup.log 2>&1" | sudo tee /etc/cron.d/observal-ch-backup ``` See [Backup and restore](/self-hosting/backup-and-restore) for detailed restore procedures. ## Step 8: First login Install the CLI on your local machine: ```bash curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash ``` Log in: ```bash observal auth login # Server URL: https://observal.yourcompany.com # Email: (create your admin account or use demo creds if you kept them) ``` Verify: ```bash observal auth whoami observal auth status ``` ## Upgrades ```bash cd ~/Observal git fetch --tags git checkout v1.5.0 # or whatever version docker compose -f docker/docker-compose.yml up -d --build ``` The Compose init service applies migrations before API startup. See [Upgrades](/self-hosting/upgrades) for rollback procedures. Or use the CLI: ```bash observal server upgrade --version 1.5.0 --force --output json ``` ## Monitoring Prometheus and Grafana are optional. Start the core stack with Prometheus only: ```bash docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up -d ``` Start Prometheus and Grafana: ```bash COMPOSE_PROFILES=grafana docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up -d ``` Access Grafana at `http://your-server:3001` when the Grafana profile is enabled. For basic alerting without Grafana, add a health check cron: ```bash echo "*/5 * * * * root curl -fsS http://localhost/health > /dev/null || echo 'Observal health check failed' | mail -s 'ALERT: Observal down' ops@yourcompany.com" | sudo tee /etc/cron.d/observal-health ``` ## Security hardening The [security assurance case](/security/assurance-case) describes the deployment trust boundaries and residual risks. Before exposing to the internet: * [ ] Remove all `DEMO_*` env vars from `.env` * [ ] Set a strong `SECRET_KEY` * [ ] Restrict SSH access (key-only, no password auth) * [ ] Configure a firewall (`ufw allow 80,443/tcp && ufw enable`) * [ ] Set up [SSO](/self-hosting/authentication) if available * [ ] Enable automatic security updates (`sudo apt install unattended-upgrades`) * [ ] Bind database ports to localhost only (already done in `docker-compose.yml` via `127.0.0.1:` prefix) ## Scaling up When you outgrow a single node: | Symptom | Fix | | ------------------------------- | --------------------------------------------------------------------------------------------- | | API response times increasing | Increase `API_WORKERS` in `.env` (default 2), or bump to a bigger VM | | ClickHouse queries slow | Increase `CLICKHOUSE_MEMORY_LIMIT` or move to a bigger VM | | Disk filling up | Reduce `DATA_RETENTION_DAYS`, add a bigger disk, or move ClickHouse data to a separate volume | | Need HA / zero downtime deploys | Migrate to the [Terraform module](/self-hosting/aws-terraform) | ## Next * [Configuration](/self-hosting/configuration) — all environment variables * [Backup and restore](/self-hosting/backup-and-restore) — detailed restore procedures * [Upgrades](/self-hosting/upgrades) — safe upgrade and rollback flow * [Troubleshooting](/self-hosting/troubleshooting) — common issues # Production deployment Deploy Observal as a distributed, highly available stack on AWS or GCP using Terraform. Best for SLA-bound deployments, teams over 50 users, and any environment where uptime and horizontal scaling matter. **End state:** a fully managed Observal install with autoscaling compute, managed databases with automated failover, encrypted secrets, centralized logging, automated backups, and HTTPS on your domain — provisioned with a single `terraform apply`. ## When to use this vs. single-node | | [Single-node](/self-hosting/single-node-deploy) | Production (this guide) | | ------------------ | ---------------------------------------------------------------- | -------------------------------------------------------------- | | **Best for** | Small/mid teams, internal use, POCs | Enterprise, SLA-bound, high-traffic | | **Infra** | 1 VM, any cloud or on-prem | \~100 managed cloud resources | | **Cost** | $20–150/mo | \~$255/mo (AWS) | | **HA** | No — single point of failure | Yes — Multi-AZ databases, autoscaling compute | | **Failover** | Manual (reboot / redeploy) | Automatic (managed services handle it) | | **Scaling** | Vertical (bigger VM) | Horizontal (add more containers) | | **Backups** | Cron + S3 | Automated (RDS snapshots, systemd timers, S3 lifecycle) | | **Secrets** | Server-package files, or direct `.env` values for source Compose | SSM Parameter Store / Secret Manager (encrypted, auditable) | | **Logging** | `docker compose logs` | CloudWatch / Cloud Logging (centralized, retained, searchable) | | **Time to deploy** | 10 minutes | 20–30 minutes | ## Choose your cloud Observal ships Terraform modules for both AWS and GCP. They provision equivalent architectures using each cloud's native managed services. ### AWS ```mermaid flowchart TB internet[Internet] alb[ALB HTTPS] api["ECS Fargate: api - 2 to 10 tasks"] web["ECS Fargate: web - 2 to 6 tasks"] worker["ECS Fargate: worker - 1 to 5 tasks"] data[EC2 data host] pg["RDS Postgres - Multi-AZ"] redis["ElastiCache Redis - 2-node"] ch["ClickHouse - EBS gp3"] grafana[Grafana] prometheus[Prometheus] internet --> alb alb -->|/api/*| api alb -->|default| web alb -->|/grafana/*| data api --> worker api --> pg api --> redis api --> ch worker --> pg worker --> redis worker --> ch data --> ch data --> grafana data --> prometheus ``` | Component | AWS Service | Notes | | ------------------- | ------------------- | ---------------------------------------------- | | Compute (stateless) | ECS Fargate | api, web, worker as separate services | | Postgres | RDS Postgres 16 | Multi-AZ on prod, encrypted, automated backups | | Redis | ElastiCache Redis 7 | 2-node replication, automatic failover on prod | | ClickHouse | EC2 + EBS gp3 | Self-hosted | | Load balancer | ALB | HTTPS via ACM, path-based routing | | Secrets | SSM Parameter Store | Encrypted with KMS, injected into ECS tasks | | Logging | CloudWatch | Per-service log groups | | Backups | S3 + RDS snapshots | Lifecycle: Standard → IA → Glacier → expire | | DNS | Route 53 | Optional; works without a custom domain | **Full walkthrough:** [AWS deployment with Terraform](/self-hosting/aws-terraform) **Quick start:** ```bash git clone https://github.com/Observal/Observal.git cd Observal/infra/terraform/aws # Configure cp terraform.tfvars.example terraform.tfvars # Edit: region, environment, domain_name (optional), alb_ingress_cidrs # Deploy terraform init terraform plan -out tf.plan terraform apply tf.plan # Verify terraform output app_url curl -fsS "$(terraform output -raw app_url)/readyz" ``` Apply takes ~~20–30 minutes (RDS Multi-AZ provisioning dominates). Baseline cost: \*\*~~$255/mo\*\* in us-east-1. ### GCP ```mermaid flowchart TB internet[Internet] lb["Global HTTPS LB - managed SSL"] api[Cloud Run: api] web[Cloud Run: web] init["Cloud Run Job: init - migrations"] worker[Cloud Run: worker] data["GCE data host - IAP"] pg["Cloud SQL - Postgres"] redis["Memorystore - Redis"] ch["ClickHouse - Persistent Disk"] grafana[Grafana] prometheus[Prometheus] internet --> lb lb -->|/api/*| api lb -->|default| web lb -->|/grafana/*| data init --> pg api --> worker api --> pg api --> redis api --> ch worker --> pg worker --> redis worker --> ch data --> ch data --> grafana data --> prometheus ``` | Component | GCP Service | Notes | | ------------------- | --------------- | ------------------------------------- | | Compute (stateless) | Cloud Run v2 | api, web, worker as separate services | | Migrations | Cloud Run Jobs | One-shot init task | | Postgres | Cloud SQL | HA configuration available | | Redis | Memorystore | Managed Redis | | ClickHouse | GCE instance | Docker Compose on a single VM | | Load balancer | Global HTTPS LB | Managed SSL certificate | | Secrets | Secret Manager | Injected into Cloud Run at start | | Logging | Cloud Logging | Built-in, no config needed | | Backups | GCS | Versioned bucket with lifecycle | | DNS | Cloud DNS | Optional | | Shell access | IAP SSH tunnel | No public SSH, no SSH keys | **Full walkthrough:** [GCP deployment with Terraform](/self-hosting/gcp-terraform) **Quick start:** ```bash cd Observal/infra/terraform/gcp # Enable required APIs gcloud services enable \ run.googleapis.com sqladmin.googleapis.com redis.googleapis.com \ compute.googleapis.com secretmanager.googleapis.com \ dns.googleapis.com vpcaccess.googleapis.com \ servicenetworking.googleapis.com # Configure cp terraform.tfvars.example terraform.tfvars # Edit: project_id, region, domain_name (optional) # Deploy terraform init terraform plan -out tf.plan terraform apply tf.plan # Run migrations gcloud run jobs execute observal-prod-init --region=us-central1 # Verify terraform output app_url ``` ## Day-2 operations These apply to both AWS and GCP. Cloud-specific commands are in the respective guides. ### Upgrade to a new release ```hcl # terraform.tfvars image_tag = "v1.5.0" ``` ```bash terraform apply ``` The init task re-runs migrations. Compute services roll over with zero downtime. ### Roll back Set `image_tag` to the previous version and `terraform apply`. Database data is not affected. If the failed version ran a destructive migration (rare, always called out in the CHANGELOG), restore from the pre-upgrade database backup. ### Scale up Adjust variables in `terraform.tfvars`: ```hcl # More API containers api_desired_count = 4 api_autoscale_max = 20 # Bigger database db_instance_class = "db.m6g.large" # AWS # postgres_tier = "db-custom-4-16384" # GCP # Bigger ClickHouse host data_instance_type = "m6i.xlarge" # AWS # data_machine_type = "e2-standard-4" # GCP ``` ```bash terraform apply ``` ### Tear down ```bash terraform destroy ``` On prod, databases have deletion protection enabled. Disable manually before destroy if you really mean it. ## Cost comparison ### AWS (us-east-1, on-demand) | Component | Spec | \~$/month | | --------------------- | ------------------ | ------------ | | Fargate api 2× | 0.5 vCPU / 1 GB | $30 | | Fargate web 2× | 0.25 vCPU / 0.5 GB | $15 | | Fargate worker 1× | 0.5 vCPU / 1 GB | $15 | | EC2 data host | t3.large | $60 | | RDS Postgres Multi-AZ | db.t4g.small | $50 | | ElastiCache Redis 2× | cache.t4g.micro | $25 | | ALB | — | $20 | | NAT Gateway | — | $33 + egress | | EBS gp3 100 GB | — | $8 | | S3 backups | \~1 GB cold | $0.10 | | **Total** | | **\~$255** | Set `environment = "staging"` to halve the bill (single-AZ RDS, one Redis node, no deletion protection). ### GCP (us-central1, on-demand) | Component | Spec | \~$/month | | ---------------------- | ------------------------------- | ---------- | | Cloud Run api | 1 vCPU / 512 MB, min 1 instance | $25 | | Cloud Run web | 1 vCPU / 256 MB, min 1 instance | $15 | | Cloud Run worker | 1 vCPU / 512 MB, min 1 instance | $25 | | Cloud SQL Postgres | db-f1-micro (shared) | $10 | | Memorystore Redis | 1 GB basic | $35 | | GCE data host | e2-standard-2 | $50 | | Global HTTPS LB | — | $18 | | Persistent Disk 100 GB | — | $4 | | GCS backups | \~1 GB cold | $0.02 | | **Total** | | **\~$180** | GCP is typically cheaper due to Cloud Run's per-request billing and no NAT Gateway equivalent charge. ## Production hardening checklist Applies to both clouds. See the cloud-specific guides for implementation details. * [ ] Enable remote Terraform state (S3 + DynamoDB on AWS, GCS on GCP) * [ ] Restrict load balancer ingress to known CIDRs * [ ] Enable cloud security services (GuardDuty + Config on AWS, Security Command Center on GCP) * [ ] Set up alerting on database CPU, memory, and 5xx error rates * [ ] Attach a WAF to the load balancer * [ ] Enable Redis transit encryption * [ ] Configure [SSO](/self-hosting/authentication) (SAML or OIDC) * [ ] Test the [backup and restore](/self-hosting/backup-and-restore) procedure end-to-end * [ ] Replace the GitHub tarball pull in the data host bootstrap with an artifact you control ## Next * [AWS deployment with Terraform](/self-hosting/aws-terraform) — full AWS walkthrough * [GCP deployment with Terraform](/self-hosting/gcp-terraform) — full GCP walkthrough * [Configuration](/self-hosting/configuration) — environment variables reference * [Upgrades](/self-hosting/upgrades) — upgrade and rollback procedures * [Backup and restore](/self-hosting/backup-and-restore) — detailed restore procedures # Docker Compose setup Step-by-step bring-up of the Observal stack. End state: the core services are healthy, API responding at `http://localhost/health`, web UI at `http://localhost`. Prometheus and Grafana are optional. ## 1. Clone and configure ```bash git clone https://github.com/Observal/Observal.git cd Observal cp .env.example .env ``` The `.env.example` ships with working direct-value defaults for local source development, including demo account credentials. You do not need to edit it for local development. Production server-package installs use generated files under `secrets/` instead; see [Configuration](/self-hosting/configuration#secret-files). > \[!NOTE] You need Docker Engine ≥ 24.0 with Compose v2 (`docker compose`, not `docker-compose`). Homebrew's Docker formula is outdated. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages. Verify with `docker version` and `docker compose version`. ## 2. Start the stack Core stack only: ```bash docker compose -f docker/docker-compose.yml up --build -d ``` With Prometheus only: ```bash docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up --build -d ``` With Prometheus and Grafana: ```bash COMPOSE_PROFILES=grafana docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up --build -d ``` First build takes a few minutes (pulls images, builds `observal-api` and `observal-web`). Subsequent starts are fast. ## 3. Verify health ```bash docker compose -f docker/docker-compose.yml ps ``` Every service should show `healthy` or `running`. The API waits for Postgres, ClickHouse, and Redis to pass health checks before starting. Expect 15–30 seconds on first boot. Hit the health endpoint: ```bash curl http://localhost/health # {"status":"ok"} ``` ## 4. Configure TLS (production only) For local dev, `http://localhost` is fine. For production, put a TLS-terminating reverse proxy in front of the nginx LB. See [Requirements → TLS / HTTPS](/self-hosting/requirements#tls--https). ## 5. Bootstrap the first user ### Option A - demo accounts (fastest for trying it out) `.env.example` seeds four demo accounts on first startup: | Role | Email | Password | | ----------- | ----------------------- | ------------------- | | Super Admin | `super@demo.example` | `super-changeme` | | Admin | `admin@demo.example` | `admin-changeme` | | Reviewer | `reviewer@demo.example` | `reviewer-changeme` | | User | `user@demo.example` | `user-changeme` | Log in with the CLI: ```bash curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash # if you haven't already observal auth login # Email: super@demo.example, Password: super-changeme ``` **Remove demo accounts before real deployment.** Unset the `DEMO_*` env vars in `.env` and restart. Already-seeded accounts stay until you delete them manually (`observal admin delete-user `). ### Option B - fresh bootstrap (recommended for production) Remove `DEMO_*` from `.env` and start the stack. Run: ```bash observal auth login # Server URL: http://localhost # No users detected - bootstrapping admin account. # Email: alice@your-company.com # Password: ************** ``` The CLI detects that no users exist and interactively creates the first admin. The `/api/v1/auth/bootstrap` endpoint is restricted to localhost access for security. ## 6. Verify with the CLI ```bash observal auth whoami observal auth status observal registry mcp list # empty list - you haven't added anything yet ``` ## 7. Stop, restart, rebuild ```bash # Stop core and any optional monitoring containers make down # Stop and delete all data, including optional monitoring volumes docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml --profile grafana down -v # Restart one service docker compose -f docker/docker-compose.yml restart observal-api # Rebuild after code changes docker compose -f docker/docker-compose.yml up --build -d observal-api ``` Makefile shortcuts from the repo root: ```bash make logs # tail core service logs make rebuild # rebuild and restart core services make up-prometheus # start core services with Prometheus make up-observability # start core services with Prometheus and Grafana make rebuild-prometheus # rebuild core services with Prometheus make rebuild-observability # rebuild core services with Prometheus and Grafana ``` ## 8. Logs ```bash docker compose -f docker/docker-compose.yml logs -f # all docker compose -f docker/docker-compose.yml logs -f observal-api # one service ``` ## 9. Port conflicts If `docker compose up` fails with `port is already allocated`, remap host ports via env vars: ```bash POSTGRES_HOST_PORT=5433 REDIS_HOST_PORT=6380 \ docker compose -f docker/docker-compose.yml up --build -d ``` Every host port is configurable. See [Ports and volumes](/self-hosting/ports-and-volumes) for the full list. ## Next → [Configuration](/self-hosting/configuration): which env vars to change for production. # AWS deployment with Terraform End state: an Observal install running in your own AWS account, fronted by an Application Load Balancer with HTTPS, with managed Postgres + Redis, ECS Fargate for the stateless app tier, and a single EC2 host for ClickHouse. Prometheus and Grafana are optional through `observability_stack`. This is the recommended path for enterprise self-hosting on AWS. If you only want to evaluate Observal, use [Docker Compose setup](/self-hosting/docker-compose) instead. ## What gets provisioned A single `terraform apply` creates: * **VPC** with public + private subnets across two availability zones, NAT gateway, VPC flow logs * **Application Load Balancer** with HTTPS (ACM certificate, DNS-validated) when you supply a domain; HTTP-only otherwise. Path-based rules: `/api/*` → api service, optional `/grafana/*` → Grafana, default → web * **ECS Fargate cluster** running: * `api` (FastAPI): 2 tasks by default, autoscales 2–10 on CPU * `web` (Next.js): 2 tasks by default, autoscales 2–6 on CPU * `worker` (arq background jobs): 1 task by default, autoscales 1–5 on CPU * `init` (one-shot migrations + seeds): runs as a Fargate `RunTask` whenever `image_tag` changes * **RDS Postgres 16**: Multi-AZ on `prod`, encrypted, automated daily backups, Performance Insights, Enhanced Monitoring, log exports * **ElastiCache Redis 7**: 2-node replication group with automatic failover on `prod`, slow-log to CloudWatch * **Data tier EC2** (Amazon Linux 2023): single host running ClickHouse on EBS gp3, optional Prometheus and Grafana, ENI with static private IP, internal Route 53 zone for DNS, daily ClickHouse → S3 snapshot via systemd timer * **S3 backups bucket**: versioned, AES256, lifecycle to STANDARD\_IA → GLACIER\_IR → expire, TLS-only * **CloudWatch log groups**: per ECS service, data host, RDS, Redis slow log, VPC flow logs * **SSM Parameter Store**: generated DB / ClickHouse / SECRET\_KEY / optional Grafana passwords, plus pre-built connection URLs injected into ECS tasks * **SSM Session Manager**: shell access to the data host, no SSH ClickHouse runs on EC2. The data volume keeps it durable across instance replacements; ClickHouse HA is out of scope for this module. ## Prerequisites | Requirement | Why | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | AWS account with billing enabled | obvious | | Terraform ≥ 1.6 | `brew install terraform` or use [tenv](https://tofuutils.github.io/tenv/) | | AWS CLI v2 | `brew install awscli` - also used by the one-shot init task runner | | [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) | shell access into the data host | | IAM principal with sufficient rights | see [Required IAM permissions](#required-iam-permissions) | | (Optional) Route 53 hosted zone | required for HTTPS on a custom domain | | (Recommended) S3 bucket + DynamoDB table | remote Terraform state - see [Remote state](#remote-state) | ## Quickstart ```bash git clone https://github.com/Observal/Observal.git cd Observal/infra/terraform/aws # 1. Authenticate to AWS (any of these works) export AWS_PROFILE=observal-prod # or: aws configure sso # or: export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... # 2. Configure cp terraform.tfvars.example terraform.tfvars $EDITOR terraform.tfvars # 3. Apply terraform init terraform plan -out tf.plan terraform apply tf.plan ``` First-time apply takes 12–15 minutes. RDS provisioning dominates. When it finishes: ```bash terraform output app_url ``` Open that URL in your browser. The API and web tasks start a couple of minutes after the ALB targets register; refresh until you see the Observal login page. A ready-to-apply call of the module lives at [`infra/terraform/aws/examples/minimal`](https://github.com/Observal/Observal/blob/main/infra/terraform/aws/examples/minimal/README.md). ## Configuration All inputs live in `terraform.tfvars`. The defaults are production-shaped; you can apply with very little changed. ### Minimal configuration (HTTP, no custom domain) ```hcl region = "us-east-1" environment = "prod" ``` The install comes up on the ALB's AWS-assigned hostname (e.g. `observal-prod-alb-1234.us-east-1.elb.amazonaws.com`) over plain HTTP. Useful for evaluation; not for production. ### Recommended configuration (HTTPS on your domain) ```hcl region = "us-east-1" environment = "prod" sizing = "medium" domain_name = "observal.example.com" route53_zone_id = "Z0123456789ABCDEFGHIJ" # Lock down ingress to your corporate egress / VPN alb_ingress_cidrs = ["203.0.113.0/24", "198.51.100.0/24"] ``` Terraform requests an ACM certificate, validates it via DNS records in your hosted zone, attaches it to the ALB, and creates the alias A record pointing the domain at the ALB. There is no manual DNS step. ### Sizing presets Instead of tuning 12+ resource variables, pick a preset: ```hcl sizing = "medium" # small | medium | large | custom ``` | Preset | \~$/mo | API tasks | Web tasks | Worker tasks | Data host | RDS | Redis | | -------- | ------ | -------------- | -------------- | -------------- | ------------------- | ------------ | --------------- | | `small` | $150 | 1× (0.25 vCPU) | 1× (0.25 vCPU) | 1× (0.25 vCPU) | t3.medium (50 GB) | db.t4g.micro | cache.t4g.micro | | `medium` | $255 | 2× (0.5 vCPU) | 2× (0.25 vCPU) | 1× (0.5 vCPU) | t3.large (100 GB) | db.t4g.small | cache.t4g.micro | | `large` | $600 | 3× (1 vCPU) | 3× (0.5 vCPU) | 2× (1 vCPU) | r6i.xlarge (500 GB) | db.r6g.large | cache.r6g.large | **Important: Presets vs. individual variables** Presets and individual resource variables (`api_cpu`, `db_instance_class`, etc.) are **mutually exclusive**: * When `sizing = "small|medium|large"`: all individual resource variables are **ignored**. The preset values are used unconditionally. * When `sizing = "custom"`: individual variables take effect normally. If you set `sizing = "medium"` and also `api_cpu = 1024`, the medium preset wins — `api_cpu` has no effect. To override individual values, set `sizing = "custom"` first. ### Custom sizing Set `sizing = "custom"` to control each resource independently: ```hcl sizing = "custom" api_cpu = 512 # 0.5 vCPU; raise to 1024+ for sustained >100 req/s api_memory = 1024 api_desired_count = 2 api_autoscale_max = 10 web_desired_count = 2 worker_desired_count = 1 # Data tier (ClickHouse) data_instance_type = "t3.large" # 8 GB - minimum viable for ClickHouse data_volume_size_gb = 100 observability_stack = "none" # none | prometheus | grafana db_instance_class = "db.t4g.small" redis_node_type = "cache.t4g.micro" ``` For high-throughput installs (>100 trace events/sec sustained), bump `data_instance_type` to `m6i.xlarge` and `db_instance_class` to `db.m6g.large`. ### Application options ```hcl deployment_mode = "enterprise" # SSO-only login data_retention_days = 90 # ClickHouse TTL; 0 to disable log_retention_days = 30 # CloudWatch log group retention image_tag = "v1.4.0" # specific Observal release; "latest" pulls main ``` See [Configuration](/self-hosting/configuration) for the meaning of each application setting. ### Bring Your Own VPC Deploy Observal into an existing VPC rather than having Terraform create a new one. Use this when you have Transit Gateway, shared-services VPCs, compliance requirements, or peered networks. ```hcl # Use an existing VPC vpc_id = "vpc-0abc1234def56789a" private_subnet_ids = ["subnet-aaa11111", "subnet-bbb22222"] public_subnet_ids = ["subnet-xxx11111", "subnet-yyy22222"] # Control ALB placement independently alb_scheme = "internet-facing" # or "internal" ``` When `vpc_id` is set, Terraform skips creating VPC, subnets, IGW, NAT gateway, route tables, and VPC flow logs. All other resources (ALB, ECS, RDS, Redis, data host) are created inside your existing VPC. **VPC requirements:** * DNS support enabled (`enableDnsSupport = true`, `enableDnsHostnames = true`) * Private subnets must have outbound internet access (via NAT Gateway or Transit Gateway) for container image pulls (`ghcr.io`) and AWS API endpoints (SSM, CloudWatch, ECR) * Public subnets must have an Internet Gateway route (only needed if `alb_scheme = "internet-facing"`) * Subnets should span at least 2 AZs for RDS Multi-AZ and ECS placement **Optional: Bring your own security groups:** For environments with strict firewall policies, supply pre-created SG IDs: ```hcl alb_security_group_id = "sg-0abc1234def56789a" ecs_security_group_id = "sg-0abc1234def56789b" ``` The ALB SG must allow inbound TCP 80/443 from your desired CIDRs. The ECS SG must allow inbound TCP 8000 and 3000 from the ALB SG, with outbound to all. A full working example lives at [`infra/terraform/aws/examples/byovpc`](https://github.com/Observal/Observal/blob/main/infra/terraform/aws/examples/byovpc/README.md). ## Required IAM permissions The IAM principal running Terraform needs permission to manage resources across these services. The simplest path is to attach the AWS-managed policies below; for a tighter custom policy, see [Hardened IAM policy](#hardened-iam-policy). | Service | Managed policy | | ----------------------------------- | --------------------------------- | | VPC, EC2 | `AmazonEC2FullAccess` | | ECS | `AmazonECS_FullAccess` | | RDS | `AmazonRDSFullAccess` | | ElastiCache | `AmazonElastiCacheFullAccess` | | Load balancer | `ElasticLoadBalancingFullAccess` | | Certificates | `AWSCertificateManagerFullAccess` | | DNS | `AmazonRoute53FullAccess` | | IAM (creates ECS + EC2 + RDS roles) | `IAMFullAccess` | | Parameter Store + Session Manager | `AmazonSSMFullAccess` | | S3 (backups bucket) | `AmazonS3FullAccess` | | Logs | `CloudWatchLogsFullAccess` | | Application Auto Scaling | `AutoScalingFullAccess` | ## Operating the install ### Tail an ECS service ```bash aws logs tail /aws/ecs/observal-prod/api --follow ``` Or get all log group names: ```bash terraform output log_group_names ``` ### Force a rolling deploy ```bash aws ecs update-service \ --cluster $(terraform output -raw ecs_cluster_name) \ --service observal-prod-api \ --force-new-deployment ``` ### Re-run migrations / seeds The init task runs automatically on every `image_tag` change. To run it manually: ```bash $(terraform output -raw init_run_task_command) ``` ### Shell into the data host ```bash $(terraform output -raw data_host_ssm_session_command) # inside the session: sudo journalctl -u observal-bootstrap -f # cloud-init / first-boot logs sudo docker compose -f /opt/observal/docker-compose.data.yml ps # CH/Grafana/Prom status sudo docker compose -f /opt/observal/docker-compose.data.yml logs -f clickhouse ``` ### Read a generated secret ```bash aws ssm get-parameter --with-decryption \ --name /observal-prod/SECRET_KEY \ --query Parameter.Value --output text ``` The Terraform run generates and stores `DB_PASSWORD`, `CLICKHOUSE_PASSWORD`, `SECRET_KEY`, and `GRAFANA_ADMIN_PASSWORD`, plus pre-built `DATABASE_URL`, `REDIS_URL`, and `CLICKHOUSE_URL` connection strings. ECS injects these into tasks at start; you never paste them in. ### Upgrade to a new Observal release ```hcl # terraform.tfvars image_tag = "v1.5.0" ``` ```bash terraform apply ``` The init task re-runs migrations against the live RDS, then ECS rolls each service over with zero downtime. ### Resize the data volume ```hcl # terraform.tfvars data_volume_size_gb = 250 ``` ```bash terraform apply # then SSM into the data host: sudo growpart /dev/nvme1n1 1 || true sudo resize2fs /dev/nvme1n1 ``` ### Destroy ```bash terraform destroy ``` In `prod`, RDS has `deletion_protection = true` and `skip_final_snapshot = false`. Disable both manually before destroy if that is really what you want. The S3 backups bucket also blocks destroy unless `backup_bucket_force_destroy = true`. ## Remote state By default, Terraform writes state to your laptop (`terraform.tfstate`). For a real install, configure remote state so the install can be managed by anyone on your team and state isn't lost. 1. Create an S3 bucket (versioned, encrypted) and a DynamoDB table with hash key `LockID`. 2. Uncomment and fill the backend block in `versions.tf`: ```hcl backend "s3" { bucket = "your-tf-state-bucket" key = "observal/prod/terraform.tfstate" region = "us-east-1" dynamodb_table = "your-tf-lock-table" encrypt = true } ``` 3. Re-run `terraform init` and answer "yes" when prompted to migrate state. ## Cost expectations Rough monthly baseline in `us-east-1` at on-demand rates (May 2026): | Component | \~$/month | | ----------------------------------- | ------------ | | Fargate api 2× (0.5 vCPU / 1 GB) | $30 | | Fargate web 2× (0.25 vCPU / 0.5 GB) | $15 | | Fargate worker 1× (0.5 vCPU / 1 GB) | $15 | | EC2 `t3.large` (data host) | $60 | | RDS `db.t4g.small` Multi-AZ | $50 | | ElastiCache (2× `cache.t4g.micro`) | $25 | | ALB | $20 | | NAT Gateway | $33 + egress | | EBS gp3 100 GB | $8 | | S3 backups (1 GB cold) | $0.10 | | **Baseline** | **\~$255** | Set `environment = "staging"` to drop RDS to single-AZ, run a single Redis node, and skip RDS deletion protection, typically halves the bill. Drop `worker_desired_count` and `web_desired_count` for further savings. ## Production hardening checklist The defaults are safe but conservative. Before pointing real traffic at this: * [ ] Switch to remote state (S3 + DynamoDB) * [ ] Restrict `alb_ingress_cidrs` to known networks * [ ] Enable AWS GuardDuty + Config in the account * [ ] Add CloudWatch alarms on RDS `CPUUtilization`, `FreeableMemory`, ECS service CPU, ALB `HTTPCode_Target_5XX_Count` * [ ] Attach AWS WAF to the ALB * [ ] Set `transit_encryption_enabled = true` on the ElastiCache replication group and switch `REDIS_URL` to `rediss://...` * [ ] Configure Observal SSO. See [Authentication and SSO](/self-hosting/authentication) * [ ] Test the [backup and restore](/self-hosting/backup-and-restore) procedure end-to-end * [ ] Replace the GitHub tarball download in `user-data.sh.tftpl` with an artifact URL you control ## Troubleshooting **`terraform apply` succeeds but the URL returns 502.** The api/web tasks are still starting. Watch the service: ```bash aws ecs describe-services \ --cluster $(terraform output -raw ecs_cluster_name) \ --services observal-prod-api \ --query 'services[0].deployments' aws logs tail /aws/ecs/observal-prod/api --follow ``` The `null_resource.run_init` step also runs migrations before the api service comes up. If the init task fails, the api service won't start. **Init task fails.** Check the `/aws/ecs/observal-prod/init` log group. The most common failures are: * RDS not yet reachable (transient on first apply; re-running fixes it) * Migration error (look at the entrypoint output) To re-run by hand: `$(terraform output -raw init_run_task_command)`. **ALB target health is `unhealthy`.** * For `api`: health check hits `/readyz` on port 8000. Tail the api log group for startup errors. * For `web`: health check hits `/` on port 3000. Check the web log group. * For Grafana: health check hits `/api/health` on port 3001. SSM into the data host and check `docker compose ps`. **ACM certificate stuck in `Pending validation`.** The DNS validation records were not created in your hosted zone. Verify `route53_zone_id` is correct and that the IAM principal had `route53:ChangeResourceRecordSets` permission at apply time. Re-run `terraform apply` after fixing. **RDS storage is full.** `max_allocated_storage` autoscales up to 500 GB by default. Raise `db_max_allocated_storage_gb` if you have heavier audit-log volume. For application-level issues (login fails, traces missing) see [Troubleshooting](/self-hosting/troubleshooting). # GCP deployment with Terraform End state: Observal running in your GCP project on Cloud Run, backed by Cloud SQL Postgres, Memorystore Redis, and a GCE instance for ClickHouse — with a Global HTTPS Load Balancer, managed SSL certificate, Secret Manager for credentials, and GCS for backups. For the overall deployment strategy and comparison with AWS, see [Production deployment](/self-hosting/production-deploy). If you want a simpler single-VM setup, see [Single-node deployment](/self-hosting/single-node-deploy). ## What gets provisioned A single `terraform apply` creates: * **VPC** with a private subnet, Cloud NAT, and a Serverless VPC Access Connector * **Cloud Run v2 services**: `api`, `web`, `worker` with autoscaling (min/max instance counts) * **Cloud Run v2 job**: `init` (one-shot migrations + seeds) * **Cloud SQL Postgres** (Open-source distribution): optional HA, encrypted, automated backups * **Memorystore Redis**: BASIC or STANDARD\_HA tier * **GCE instance** (data host): ClickHouse on a persistent disk, with optional Prometheus and Grafana, accessible via IAP SSH tunnel * **Global HTTPS Load Balancer** with managed SSL certificate (when domain is supplied) * **Cloud DNS** A record pointing to the load balancer * **Secret Manager**: generated DB / ClickHouse / SECRET\_KEY passwords, plus connection URLs injected into Cloud Run services * **GCS backups bucket**: versioned, lifecycle to Nearline → delete * **Artifact Registry**: for storing container images (optional) ## Prerequisites | Requirement | Why | | --------------------------------- | --------------------------------------- | | GCP project with billing enabled | Resources cost money | | `gcloud` CLI authenticated | `gcloud auth application-default login` | | Terraform ≥ 1.5 | `brew install terraform` | | (Optional) Cloud DNS managed zone | Required for HTTPS on a custom domain | Enable required APIs: ```bash gcloud services enable \ run.googleapis.com \ sqladmin.googleapis.com \ redis.googleapis.com \ compute.googleapis.com \ secretmanager.googleapis.com \ dns.googleapis.com \ vpcaccess.googleapis.com \ servicenetworking.googleapis.com \ iam.googleapis.com ``` ## Quickstart ```bash git clone https://github.com/Observal/Observal.git cd Observal/infra/terraform/gcp # 1. Authenticate gcloud auth application-default login # 2. Configure cp terraform.tfvars.example terraform.tfvars $EDITOR terraform.tfvars # At minimum, set project_id # 3. Apply terraform init terraform plan -out tf.plan terraform apply tf.plan # 4. Run migrations $(terraform output -raw init_job_run_command) # 5. Verify terraform output app_url ``` ## Configuration All inputs live in `terraform.tfvars`. ### Minimal (no custom domain) ```hcl project_id = "my-gcp-project" region = "us-central1" environment = "prod" ``` The install comes up on the Cloud Run default URL (e.g. `https://observal-prod-api-xxxxx-uc.a.run.app`). ### Recommended (HTTPS on your domain) ```hcl project_id = "my-gcp-project" region = "us-central1" environment = "prod" domain_name = "observal.example.com" dns_managed_zone_name = "example-com" ``` Terraform provisions a Global HTTPS Load Balancer with a Google-managed SSL certificate and creates the DNS A record. ### Sizing ```hcl # Cloud Run api_cpu = "1" api_memory = "1Gi" api_min_instances = 1 api_max_instances = 10 web_min_instances = 1 worker_min_instances = 1 # Data tier data_machine_type = "e2-standard-2" # 2 vCPU / 8 GB data_disk_size_gb = 100 # Cloud SQL db_tier = "db-g1-small" db_disk_size_gb = 50 db_ha_enabled = false # set true for production HA # Redis redis_memory_size_gb = 1 redis_tier = "BASIC" # or STANDARD_HA ``` For high-throughput installs, bump `data_machine_type` to `e2-standard-4`, `db_tier` to `db-custom-4-16384`, and `redis_tier` to `STANDARD_HA`. ## Operating the install ### Shell into the data host ```bash $(terraform output -raw data_host_ssh_command) # Inside: sudo docker compose -f /opt/observal/docker-compose.data.yml ps sudo docker compose -f /opt/observal/docker-compose.data.yml logs -f clickhouse ``` No public SSH. Access is through IAP (Identity-Aware Proxy) — authenticated, audited, no SSH keys to manage. ### View logs Cloud Run logs go to Cloud Logging automatically: ```bash gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=observal-prod-api" \ --limit 50 --format json ``` Or use the Cloud Console: **Logging → Logs Explorer**, filter by `resource.type="cloud_run_revision"`. ### Re-run migrations ```bash $(terraform output -raw init_job_run_command) ``` ### Read a generated secret ```bash gcloud secrets versions access latest \ --secret="observal-prod-SECRET_KEY" ``` ### Upgrade to a new release ```hcl # terraform.tfvars image_tag = "v1.5.0" ``` ```bash terraform apply $(terraform output -raw init_job_run_command) ``` Cloud Run handles the rolling deploy. Zero downtime. ### Resize the data disk ```hcl data_disk_size_gb = 250 ``` ```bash terraform apply # SSH into the data host and resize the filesystem: $(terraform output -raw data_host_ssh_command) sudo growpart /dev/sdb 1 || true sudo resize2fs /dev/sdb1 ``` ### Destroy ```bash terraform destroy ``` Cloud SQL deletion protection is enabled on prod. Disable manually before destroy if you mean it. ## Cost estimate (us-central1, on-demand) | Component | Spec | \~$/month | | ---------------------- | ------------------------------- | ---------- | | Cloud Run api | 1 vCPU / 1 GB, min 1 instance | $25 | | Cloud Run web | 1 vCPU / 512 MB, min 1 instance | $15 | | Cloud Run worker | 1 vCPU / 1 GB, min 1 instance | $25 | | Cloud SQL Postgres | db-g1-small (shared) | $25 | | Memorystore Redis | 1 GB BASIC | $35 | | GCE data host | e2-standard-2 | $50 | | Global HTTPS LB | — | $18 | | Persistent Disk 100 GB | — | $4 | | GCS backups | \~1 GB | $0.02 | | **Total** | | **\~$200** | Cloud Run scales to zero when idle (if `min_instances = 0`), which can significantly reduce costs for low-traffic deployments. ## Production hardening checklist * [ ] Set `db_ha_enabled = true` for Cloud SQL high availability * [ ] Set `redis_tier = "STANDARD_HA"` for Redis failover * [ ] Restrict Cloud Run ingress (use `ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"` if behind GCLB) * [ ] Enable Cloud Armor (WAF) on the Global HTTPS Load Balancer * [ ] Enable Security Command Center in the project * [ ] Set up alerting on Cloud SQL CPU, memory, and Cloud Run error rates * [ ] Configure [SSO](/self-hosting/authentication) * [ ] Test [backup and restore](/self-hosting/backup-and-restore) end-to-end * [ ] Set up Terraform remote state in GCS ## Troubleshooting **Cloud Run service stuck in `Revision is not ready`.** Check revision logs in Cloud Logging. Common causes: Secret Manager permissions missing, VPC connector not ready, or image pull failure. **Init job fails.** ```bash gcloud run jobs executions list --job=observal-prod-init --region=us-central1 gcloud logging read "resource.type=cloud_run_job AND resource.labels.job_name=observal-prod-init" --limit 20 ``` Usually: Cloud SQL not yet reachable (transient on first apply), or missing secret access. **502 from the load balancer.** The managed SSL certificate takes 10–30 minutes to provision. Check: **Network Services → Load Balancing → (your LB) → Backend services → Health**. **Cloud SQL connection refused from Cloud Run.** The VPC Access Connector must be in the same region as Cloud Run. Verify `connector_cidr` doesn't overlap with your VPC subnet. For application-level issues, see [Troubleshooting](/self-hosting/troubleshooting). ## Next * [Production deployment](/self-hosting/production-deploy) — overview of both clouds * [Configuration](/self-hosting/configuration) — environment variables * [Upgrades](/self-hosting/upgrades) — upgrade and rollback procedures * [Backup and restore](/self-hosting/backup-and-restore) — restore procedures # Configuration Boot-time infrastructure settings live in `.env`. Credentials can use dedicated files. Runtime settings, including SSO, normally live in the admin UI, while file-backed sensitive settings remain only in process memory. ## Required for production Source deployments must override these before going live. Server-package setup generates the secret values and file references automatically: | Variable | Default | Why change | | ----------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------ | | `SECRET_KEY` or `SECRET_KEY_FILE` | `change-me-to-a-random-string` | Application encryption secret. Use a random value of at least 32 characters. | | `POSTGRES_PASSWORD` or `POSTGRES_PASSWORD_FILE` | `postgres` | PostgreSQL bootstrap credential. | | `CLICKHOUSE_PASSWORD` or generated hashed user config | `clickhouse` | ClickHouse credential. | | `CORS_ALLOWED_ORIGINS` | `http://localhost:3000` | Scope to your real frontend origin(s). Configure as `deployment.cors_origins` in Admin Settings. | | `deployment.frontend_url` | `http://localhost:3000` | Used for OAuth redirects and email links. Configure in Admin Settings. | The server-package installer generates these credentials in an operator-owned `secrets/` directory. Directories use mode `0750`, files use mode `0640`, and `OBSERVAL_SECRET_GID` grants the containers read access through the operator's group. Existing deployments can keep direct environment values. With a terminal, setup prompts for the frontend URL, bind address, and observability profile. Without a terminal, the same installer selects `http://localhost:3000`, loopback binding, and no optional observability stack. It prints the initial super-admin email, generated password, and password-file path after the first successful startup. Existing headless installations preserve their configuration rather than replacing it. ## Secret files For any supported credential named `NAME`, set `NAME_FILE` to a UTF-8 file path instead of placing the value in `.env`: ```dotenv SECRET_KEY_FILE=/run/secrets/secret_key DATABASE_URL_FILE=/run/secrets/database_url JWT_KEY_PASSWORD_FILE=/run/secrets/jwt_key_password GIT_CLONE_TOKEN_FILE=/run/secrets/git_clone_token OAUTH_CLIENT_SECRET_FILE=/run/secrets/oauth_client_secret INSIGHTS_API_KEY_FILE=/run/secrets/insights_api_key ``` The reader accepts files up to 64 KiB and removes one trailing newline. Setting both `NAME` and `NAME_FILE` is an error, so precedence is never ambiguous. Replace a file atomically and restart the affected API or worker processes to rotate the value without rebuilding Observal. File-backed dynamic settings are marked externally managed in the admin API. Their contents are not imported into PostgreSQL or Redis, and the admin API rejects attempts to overwrite, delete, or revoke them. Supported file-backed boot credentials include `DATABASE_URL`, `CLICKHOUSE_URL`, `REDIS_URL`, `SECRET_KEY`, `OLD_SECRET_KEY`, `JWT_KEY_PASSWORD`, `GIT_CLONE_TOKEN`, and every `DEMO_*_PASSWORD`. SSO environment imports support the same form, including OAuth client secrets, Google and GitHub OAuth secrets, the insights provider key, SAML certificates, and the SAML key-encryption password. CLI tokens support `OBSERVAL_ACCESS_TOKEN_FILE`, `OBSERVAL_API_KEY_FILE`, and `OBSERVAL_TOKEN_FILE`. PostgreSQL containers use `POSTGRES_PASSWORD_FILE`. The server package generates a hashed ClickHouse user configuration instead of putting its database password in `.env`. Service-specific subdirectories expose only the PostgreSQL password to PostgreSQL, the ClickHouse health credential to ClickHouse, and the Grafana and ClickHouse datasource credentials to Grafana. ## SSO-only mode Set `deployment.sso_only=true` in **Admin → SSO** when you want IdP-only access. Leave it `false` to keep password login available. ## Demo accounts Seeded on first startup *only* when no users exist: ```dotenv DEMO_SUPER_ADMIN_EMAIL=super@demo.example DEMO_SUPER_ADMIN_PASSWORD=super-changeme DEMO_ADMIN_EMAIL=admin@demo.example DEMO_ADMIN_PASSWORD=admin-changeme DEMO_REVIEWER_EMAIL=reviewer@demo.example DEMO_REVIEWER_PASSWORD=reviewer-changeme DEMO_USER_EMAIL=user@demo.example DEMO_USER_PASSWORD=user-changeme ``` **Remove demo account variables and their password files before a real deployment.** Existing demo users survive after removal. Delete them manually (`observal admin delete-user `). > **Admin settings warning:** If demo accounts are still active or `SECRET_KEY` is insecure, the admin Settings page will display a warning banner at the top so operators can spot and fix the issue without digging through logs. ## Database connections Source deployments may use direct values: ```dotenv DATABASE_URL=postgresql+asyncpg://postgres:postgres@observal-db:5432/observal CLICKHOUSE_URL=clickhouse://default:clickhouse@observal-clickhouse:8123/observal REDIS_URL=redis://observal-redis:6379 ``` Server-package installs use file references instead: ```dotenv DATABASE_URL_FILE=/run/secrets/database_url CLICKHOUSE_URL_FILE=/run/secrets/clickhouse_url REDIS_URL_FILE=/run/secrets/redis_url ``` Inside Docker Compose, hostnames resolve via the `observal-net` bridge (e.g. `observal-db`). Outside Docker (e.g. CLI running on host against dockerized DBs), use `localhost:`. ## OAuth / SSO Optional. Configure OIDC, SAML, and SSO-only mode in **Admin → SSO**. OIDC client changes are stored immediately, then take effect after the API restarts. Full setup in [Authentication and SSO](/self-hosting/authentication). ## Rate limiting ``` RATE_LIMIT_AUTH=10/minute # general auth endpoints RATE_LIMIT_AUTH_STRICT=5/minute # login and password reset ``` Tighten for higher-traffic deployments. ## ClickHouse retention ``` DATA_RETENTION_DAYS=90 ``` Session events older than this are removed from ClickHouse. Set to `0` to disable retention. The minimum non-zero value enforced on startup is 7. ## JWT keys ``` JWT_SIGNING_ALGORITHM=ES256 # ES256 (default) or RS256 JWT_KEY_DIR=/data/keys # persisted in the apidata volume ``` The server generates asymmetric keys on first boot and stores them in `$JWT_KEY_DIR`. **Back up this directory**: losing the keys invalidates every session. Changing `JWT_SIGNING_ALGORITHM` and restarting retires the current public key, generates the selected key type, and keeps old tokens verifiable during their normal lifetime. SAML service-provider material can be mounted independently: ```dotenv SAML_SP_PRIVATE_KEY_FILE=/run/secrets/saml_sp_private_key SAML_SP_X509_CERT_FILE=/run/secrets/saml_sp_x509_cert ``` Set both files together. They override database-generated SP material, are reported as externally managed, and cannot be regenerated or removed through the admin API. Replace both files and restart the API to rotate them. More: [Authentication and SSO](/self-hosting/authentication). ## Git operations (submission analysis) ```dotenv ALLOW_INTERNAL_URLS=false # allow internal/private Git URLs (GitLab/GHE) GIT_CLONE_TOKEN_FILE=/run/secrets/git_clone_token GIT_CLONE_TOKEN_USER=x-access-token GIT_CLONE_TIMEOUT=120 # seconds ``` `GIT_CLONE_TOKEN_USER` varies by provider: `x-access-token` for GitHub, `oauth2` or `private-token` for GitLab. ## Observal CLI (client-side) env vars Not set in `.env` on the server. These live on the CLI user's machine. | Variable | Purpose | | -------------------------------------------- | -------------------------------- | | `OBSERVAL_SERVER_URL` | Default server URL | | `OBSERVAL_ACCESS_TOKEN` / `OBSERVAL_API_KEY` | Pre-authenticate without `login` | | `OBSERVAL_TIMEOUT` | Request timeout (seconds) | Full list: [Environment variables](/reference/environment-variables). ## Next → [Ports and volumes](/self-hosting/ports-and-volumes) # Usage reporting Observal can send aggregate usage reports to the Observal-operated collector at `https://usage.observal.io/api/v1/usage-pings`. Reporting is enabled by default with a six-hour cadence, but delivery remains dormant until a super administrator configures the company name and Deployment Public URL. The collector and reporting dashboard are maintained separately in [Observal Usage](https://github.com/Observal/observal_usage). ## Enable reporting A super administrator must configure **Deployment Public URL** and then open **Admin > Settings > Usage Reporting**, which appears near the top of the settings page. Enter the company name and save the reporting configuration. The available frequencies are every six hours, daily, and weekly. Every six hours is the default. The same panel can disable reporting, change the frequency, preview the exact payload, and send a test report. Only a super administrator can change the company identity, frequency, or consent setting. Disabling the switch stops scheduled delivery immediately. It does not remove reports already received by Observal. Contact the Observal maintainers with the installation ID shown in the panel to request deletion. ## Data included * Stable, randomly generated installation ID * Company name and deployment hostname supplied by the administrator * Observal version and deployment type * Aggregate counts for users, teams, registry components, agent installations, and sessions * Aggregate session totals by harness * Distinct active-user and active-agent counts for 7- and 30-day windows * Aggregate event, prompt, tool-call, tool-result, token, cache-token, and credit totals for 7- and 30-day windows * Thirty-day session averages for duration, prompts, and tool calls * Thirty-day counts for sessions using tools or tokens, registered versus unregistered agents, top-level versus sub-agent sessions, and distinct agent/model versions * Aggregate parser-error and truncated-event counts for ingestion health * Boolean adoption signals for selected server features * Report schema version and timestamp ## Data never included Usage reports do not include names, email addresses, user IDs, agent IDs, prompts, responses, tool arguments, tool results, trace events, source code, file paths, repository names, model identifiers, IP addresses, authentication tokens, API keys, credentials, or arbitrary configuration values. Token and credit fields contain numeric totals only. Distinct model and agent-version fields contain counts, never their names or identifiers. Only the selected feature flags listed above are included. ## Delivery behavior The selected schedule uses fixed UTC boundaries: * Every six hours: 00:30, 06:30, 12:30, and 18:30 UTC * Daily: 06:30 UTC each day * Weekly: Monday at 06:30 UTC The worker evaluates due reports every six hours. If a scheduled report was missed or failed, a later worker pass retries it until the current schedule window has a successful delivery. Each pass makes up to three total delivery attempts with short backoff. A failure never blocks normal Observal operation. The last successful send, latest error, selected frequency, and next delivery time are visible in the Usage Reporting settings panel. The collector URL is fixed in production releases. `USAGE_PING_URL` exists only to direct development and isolated test deployments to a local collector. # Ports and volumes Every exposed port and persistent volume, at a glance. ## Host ports (default) | Service | Host port | Env var (to remap) | Protocol | | -------------------- | --------- | ---------------------- | ---------------- | | API | 8000 | `API_HOST_PORT` | HTTP / WebSocket | | Web UI | 3000 | `WEB_HOST_PORT` | HTTP | | Postgres | 5432 | `POSTGRES_HOST_PORT` | TCP | | ClickHouse | 8123 | `CLICKHOUSE_HOST_PORT` | HTTP | | Redis | 6379 | `REDIS_HOST_PORT` | TCP | | Prometheus, optional | 9090 | `PROMETHEUS_HOST_PORT` | HTTP | | Grafana, optional | 3001 | `GRAFANA_HOST_PORT` | HTTP | The worker has no exposed port; it talks to Redis and ClickHouse internally only. ### Remap on startup ```bash POSTGRES_HOST_PORT=5433 REDIS_HOST_PORT=6380 \ docker compose -f docker/docker-compose.yml up --build -d ``` ### Checking port conflicts ```bash # macOS / Linux lsof -nP -iTCP:5432 -sTCP:LISTEN lsof -nP -iTCP:6379 -sTCP:LISTEN # Windows (PowerShell) netstat -ano | findstr :5432 ``` ## Persistent volumes All volumes are named and managed by Docker. They survive `docker compose down` but are deleted by `docker compose down -v`. | Volume | Mount point | Contents | Loss impact | | ----------------------- | -------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------- | | `pgdata` | `/var/lib/postgresql/data` | Postgres data (users, registry, RBAC) | Catastrophic - all accounts, agents, MCPs lost | | `chdata` | `/var/lib/clickhouse` | ClickHouse session, audit, and security data | High: all telemetry lost; accounts and registry survive | | `redisdata` | `/data` | Redis persistence | Low - job queue lost; pending jobs need to be re-kicked | | `grafanadata`, optional | `/var/lib/grafana` | Grafana dashboards and config | Medium - custom dashboards lost; defaults are re-provisioned | | `apidata` | `/data` (API container) | **JWT signing keys** | Catastrophic - every session invalidated, users must re-login; backup/restore required | **Back up `apidata` and `pgdata` before any upgrade.** See [Backup and restore](/self-hosting/backup-and-restore). ## Security posture defaults The stack uses hardened defaults out of the box: * **Read-only root filesystem** on `observal-api`, `observal-worker`, `observal-web` (tmpfs for `/tmp`). * **`no-new-privileges`** on every service. * **Memory limits** on all services (see [Requirements](/self-hosting/requirements#hardware)). * **Health checks** on Postgres, ClickHouse, Redis, API. * **Bridge network** (`observal-net`) isolates inter-service traffic. ## Internal DNS Inside the compose network, services resolve each other by service name: | From | To | URL | | ------------ | ---------- | ------------------------------------------------------------------- | | API / worker | Postgres | `postgresql+asyncpg://postgres:postgres@observal-db:5432/observal` | | API / worker | ClickHouse | `clickhouse://default:clickhouse@observal-clickhouse:8123/observal` | | API / worker | Redis | `redis://observal-redis:6379` | | Web | API | `http://observal-api:8000` | The CLI running on your host machine resolves via `localhost:`. ## Next → [Databases](/self-hosting/databases) # Databases Observal runs two DBs with very different jobs. | DB | Role | Access pattern | Schema source of truth | | --------------- | --------------------------------- | --------------------------------- | -------------------------------------------------------------------- | | Postgres 16 | Registry, users, config | Relational, transactional | Alembic migrations in `observal-server/alembic/versions/` | | ClickHouse 26.5 | Telemetry and audit event storage | Columnar, time-series, high-write | Versioned SQL migrations in `observal-server/clickhouse/migrations/` | ## Postgres ### What's in it * `users`, `roles`, RBAC bindings * `mcps`, `agents`, `skills`, `hooks`, `prompts`, `sandboxes`: registry metadata * `reviews`: submission review state * `feedback`, `ratings` * `alerts`, `alert_history` * `api_keys` * `audit_log` and related audit tables ### Migrations Managed by Alembic. The server applies pending migrations automatically on startup. Migration files live in `observal-server/alembic/versions/`. For Docker Compose deployments, run the init service manually when needed: ```bash docker compose -f docker/docker-compose.yml run --rm observal-init ``` The init service applies Alembic and ClickHouse migrations before API startup. `observal server migrate` moves data between deployments; it does not apply schema migrations. ### Reset To wipe the registry and start over: ```bash docker compose -f docker/docker-compose.yml down -v docker compose -f docker/docker-compose.yml up --build -d ``` The `-v` deletes all named volumes. Use only in dev. *** ## ClickHouse ### What's in it Core tables: | Table | Contents | | -------------------- | ----------------------------------------------------------------------------------- | | `session_events` | Raw and parsed harness JSONL lines, token fields, tool fields, and session metadata | | `session_stats_agg` | Pre-aggregated session list and summary metrics from `session_events` | | `layer_snapshots` | Harness config snapshots used by version-aware insights | | `audit_log` | Audit events | | `security_events` | Security events for login, auth, and admin activity | | `webhook_deliveries` | Alert webhook delivery attempts and status | ### Deduplication and aggregates `session_events` and `layer_snapshots` use `ReplacingMergeTree` for idempotent ingest. `session_stats_agg` uses `AggregatingMergeTree` and is maintained by a materialized view. The API query layer handles the required `FINAL` or aggregate reads. If you query ClickHouse directly, match the table engine instead of assuming every table reads the same way. ### Retention (TTL) Controlled by `DATA_RETENTION_DAYS`: * Default `90`: rows older than 90 days are TTL'd out. * `0`: retention disabled (disk grows without bound). * The server enforces a minimum of `7` on any non-zero value. TTL runs asynchronously. Disk space is reclaimed on the next merge; don't expect instant free-up. ### Schema migrations ClickHouse schema changes are managed separately from Alembic. Alembic is only for Postgres. ClickHouse migration files live in: ```bash observal-server/clickhouse/migrations/*.sql ``` The init container runs ClickHouse migrations after Alembic and before the API starts. The migration runner records applied files in `clickhouse_schema_migrations`. On existing installations that predate versioned ClickHouse migrations, the runner detects the existing baseline tables and stamps `001_baseline.sql` as applied instead of replaying the whole baseline. For local checks outside Docker, run the same runner from the server package: ```bash cd observal-server python -m services.clickhouse.migrations ``` Do not put ClickHouse DDL in startup code. Add a new migration file instead. ### Capacity planning Session record size depends on harness transcript detail and tool output size. Measure representative sessions, apply the configured raw-line retention window, and plan 2 to 3 times headroom for merges and replicas. ### External ClickHouse For heavy workloads, run ClickHouse on a dedicated host outside the compose stack. Point the API at it: ``` CLICKHOUSE_URL=clickhouse://user:pass@external-clickhouse.example.com:8123/observal ``` Remove the `observal-clickhouse` service from `docker-compose.yml` or ignore it. *** ## Backup See [Backup and restore](/self-hosting/backup-and-restore). Short version: * Postgres: `pg_dump` from a running container. * ClickHouse: snapshot the `chdata` volume, or use ClickHouse's native `BACKUP` command. * Both: back up before every upgrade. ## Next → [Authentication and SSO](/self-hosting/authentication) # Authentication and SSO How Observal authenticates users and signs tokens, and how to wire up SSO. ## Authentication modes Observal supports password auth, API keys, OAuth / OIDC, and SAML. The public login page shows only the methods enabled for the deployment. | Method | How it is enabled | Notes | | ----------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | Email + password | Default password auth | Used by bootstrap admins and locally managed users | | Self registration | `auth.self_registration_enabled=true` | Creates standard `user` accounts only | | OAuth / OIDC | SSO tab settings: `oauth.client_id`, `oauth.client_secret`, `oauth.server_metadata_url` | Uses IdP discovery metadata. API restart required after changes. | | SAML | SSO tab settings: `saml.*` | SAML setup with JIT provisioning | | API keys | User generated after login | Inherits the user's role | Use `deployment.sso_only=true` when password login should be hidden and only SSO should be available. ## SSO-only mode `deployment.sso_only` controls whether password-based authentication is available. | Value | Effect | | ----------------- | ----------------------------------------------------------------------------------------------------------- | | `false` (default) | Email and password login stays available alongside SSO. | | `true` | Password login, password reset, and local user bootstrap are blocked. Users must sign in with OIDC or SAML. | Set it in **Admin → SSO → Access policy**. Confirm OIDC or SAML works with a real test user before enabling it. ## Self registration Controls whether visitors can create their own Observal account from the login page. | Value | Effect | | ----------------- | -------------------------------------------------------------------------------- | | `true` | Shows a **Register** button on the login page and allows public account creation | | `false` (default) | Hides registration and blocks `POST /api/v1/auth/register` | You can set this in the web UI at **Admin → Settings → Authentication → Self Registration Enabled**. If you prefer the CLI, set the same dynamic setting directly: ```bash observal admin set auth.self_registration_enabled true ``` New accounts are created with the built-in `user` role. They cannot review submissions, manage users, or change server settings unless an admin promotes them later. Disable it again with: ```bash observal admin set auth.self_registration_enabled false ``` ## The bootstrap flow On a fresh server with no users, the `/api/v1/auth/bootstrap` endpoint is available **to localhost only**. When you run `observal auth login`, the CLI detects the empty user table and bootstraps an admin account interactively. This is how you create the first admin without any pre-existing credential. Once the first admin exists, bootstrap is disabled. ## JWT signing keys Tokens are signed with asymmetric keys (ES256 by default, RS256 also supported). Keys are generated on first startup and stored in the `apidata` volume at `$JWT_KEY_DIR` (default `/data/keys`). ``` JWT_SIGNING_ALGORITHM=ES256 # ES256 (default) or RS256 JWT_KEY_DIR=/data/keys ``` ### Critical: back up `$JWT_KEY_DIR` Losing these keys invalidates **every** access and refresh token. All users must log in again. Tokens rotate, but only the private key can issue new ones, and there is no recovery path without the keys. Back up the `apidata` volume every time you back up Postgres. See [Backup and restore](/self-hosting/backup-and-restore). ### Key rotation and algorithm changes For a same-algorithm emergency rotation, stop the API, back up and remove `signing.pem`, then restart. New keys are generated and existing sessions must authenticate again. To move between ES256 and RS256, change `JWT_SIGNING_ALGORITHM` and restart. Observal archives the old public key, generates the selected key type, publishes both in JWKS, and continues verifying old tokens until they expire. New tokens use only the configured algorithm. Unsupported algorithms and token headers that do not match the resolved key type are rejected. If the signing key is encrypted, use `JWT_KEY_PASSWORD_FILE` and rotate the password and key together during a planned restart. ### File-backed SAML keys SAML service-provider keys can remain outside the settings database: ```dotenv SAML_SP_PRIVATE_KEY_FILE=/run/secrets/saml_sp_private_key SAML_SP_X509_CERT_FILE=/run/secrets/saml_sp_x509_cert ``` Both files are required together. File-backed material overrides the generated database key, appears as externally managed in admin responses, and cannot be replaced through the admin API. Replace both files atomically and restart the API to rotate them. OAuth, Google, GitHub, and SAML secret environment imports also accept the `NAME_FILE` form. File-backed values stay in memory rather than being copied into PostgreSQL or Redis. See [Configuration](/self-hosting/configuration#secret-files). ## OAuth / OIDC SSO Set these three in **Admin → SSO → SSO settings**, then restart the API so the OIDC client is rebuilt: | Setting | Value | | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `oauth.client_id` {#oauth-client-id} | Client ID from your IdP | | `oauth.client_secret` {#oauth-client-secret} | Client secret from your IdP | | `oauth.server_metadata_url` {#oauth-server-metadata-url} | OIDC discovery URL, for example `https://accounts.example.com/.well-known/openid-configuration` | Observal uses [Authlib](https://docs.authlib.org/) and reads the IdP discovery document, so any OIDC-compliant provider works (Auth0, Okta, Azure AD, Google Workspace, Keycloak, Authentik, Dex, etc.). ### OIDC Client ID The public client identifier from your IdP application registration. ### OIDC Client Secret The private client secret from your IdP application registration. It is stored encrypted and is never shown again after saving. ### OIDC Discovery URL The `.well-known/openid-configuration` URL for your IdP tenant or authorization server. ### Redirect URI Configure your IdP to allow: ``` {FRONTEND_URL}/api/v1/auth/oauth/callback ``` With `FRONTEND_URL=https://observal.your-company.internal`, that's: ``` https://observal.your-company.internal/api/v1/auth/oauth/callback ``` ### First OAuth login The first user who logs in via OAuth is **not** automatically an admin. Bootstrap a local admin first (via `observal auth login` before enabling OAuth, or via the demo super admin), then use that admin to promote the OAuth user. ### Scope / claims Observal requests standard `openid profile email` scope. The IdP's `email` claim is the canonical user identifier. ## Google OAuth (first-class provider) Google sign-in runs as its own provider, separate from the generic OIDC slot above. Both can be enabled at the same time, so a deployment can offer Okta *and* Google on the login screen. Set these in the SSO settings page, or set them as container env vars for one-time import at startup. The **Sign in with Google** button appears after the API restarts: ``` GOOGLE_OAUTH_CLIENT_ID=1234567890-abc...apps.googleusercontent.com GOOGLE_OAUTH_CLIENT_SECRET=GOCSPX-... ``` The Google OIDC discovery URL is hardcoded server-side, so you don't need to set it. ### Creating the Google OAuth client 1. Open the [Google Cloud Console](https://console.cloud.google.com/apis/credentials) in the project you want to use. 2. Click **Create Credentials → OAuth client ID**. 3. **Application type:** Web application. 4. **Authorized JavaScript origins:** `{FRONTEND_URL}` (e.g. `https://observal.your-company.internal`). 5. **Authorized redirect URI:** `{FRONTEND_URL}/api/v1/auth/oauth/google/callback`. 6. Copy the generated **Client ID** and **Client secret** into SSO settings or your container env. 7. Restart the API container so the Authlib client is rebuilt. ### Restricting to specific email domains Set `GOOGLE_OAUTH_ALLOWED_DOMAINS` to a comma-separated list of domains. Anyone outside the list is rejected with a 403, even if they have a valid Google account. ``` GOOGLE_OAUTH_ALLOWED_DOMAINS=acme.com,acme.io ``` Leave it unset to allow any Google account (including personal `@gmail.com` addresses) to provision themselves as `role=user`. ### Notes * Observal additionally requires Google's `email_verified` claim to be `true`. Unverified accounts (rare on Google but possible) are rejected with a 400. * The first Google user is **not** automatically an admin (matches the generic OIDC behavior). Bootstrap a local admin first, then use that account to promote the Google user. * The auth provider and Google subject ID are recorded on the user row (`auth_provider="google"`, `sso_subject_id=`) for audit purposes. ## GitHub OAuth (first-class provider) GitHub sign-in runs as its own provider, alongside generic OIDC and Google. Unlike those two, GitHub is plain OAuth 2.0 (not OIDC): there is no discovery URL and no ID token. Observal fetches the profile and email list from the GitHub REST API after the code exchange. Set these in the SSO settings page, or as container env vars for one-time import at startup. The **Sign in with GitHub** button appears after the API restarts: ``` GITHUB_OAUTH_CLIENT_ID=Iv1.abc123... GITHUB_OAUTH_CLIENT_SECRET=ghp_... / github_pat_... style secret ``` ### Creating the GitHub OAuth app 1. Open **Settings → Developer settings → OAuth Apps** (on your personal account or, preferably, your GitHub organization) and click **New OAuth App**. 2. **Homepage URL:** `{FRONTEND_URL}` (e.g. `https://observal.your-company.internal`). 3. **Authorization callback URL:** `{FRONTEND_URL}/api/v1/auth/oauth/github/callback`. 4. Copy the generated **Client ID** and generate a **Client secret**; put both into SSO settings or your container env. 5. Restart the API container so the Authlib client is rebuilt. ### Restricting to GitHub organizations Set `GITHUB_OAUTH_ALLOWED_ORGS` to a comma-separated list of GitHub organization slugs. Only *active* members of at least one listed org can sign in; everyone else is rejected with a 403. Pending invitations don't count. ``` GITHUB_OAUTH_ALLOWED_ORGS=acme-inc,acme-labs ``` When an org allowlist is configured, Observal requests the `read:org` scope (in addition to `read:user user:email`) so it can see private org memberships. Members may need to grant/request org approval for the OAuth app if the org restricts third-party access. Leave it unset to allow any GitHub account to provision itself as `role=user`. ### Notes * Observal only accepts **verified** email addresses from `GET /user/emails` (preferring the primary). The profile-level `email` field is never trusted, and accounts with no verified email are rejected with a 400. * Departments are not populated automatically from GitHub (there is no groups claim). Assign them in **Admin → Users**, individually or via bulk upload — same as Google users. * The auth provider and the *numeric* GitHub user ID are recorded on the user row (`auth_provider="github"`, `sso_subject_id=`). The numeric ID is used instead of the login handle because handles can be renamed and re-registered. ## Role-based access control (RBAC) Four built-in roles enforced on every endpoint: | Role | Typical abilities | | ------------- | ------------------------------------------------------- | | `user` | Publish components, install agents, view their own data | | `reviewer` | + approve/reject registry submissions | | `admin` | + manage users, change server settings | | `super_admin` | + sensitive super-admin-only operations | Change a user's role: ```bash observal admin users # GET /api/v1/admin/users/{id}/role to inspect # PUT /api/v1/admin/users/{id}/role to change ``` Or in the web UI at `/settings/users`. ## API keys Users can generate API keys for scripts and CI. The key inherits the user's role. ```bash # Get a key - flow depends on your setup (web UI usually) # Then in CI: export OBSERVAL_API_KEY= export OBSERVAL_SERVER_URL=https://observal.your-company.internal observal ops traces --limit 100 --output json | jq ``` Keys can be revoked via `POST /api/v1/auth/token/revoke`. ## Rate limits Auth endpoints are rate-limited to slow brute-force attempts: | Setting | Default | Scope | | ------------------------ | ----------- | --------------------------------------- | | `RATE_LIMIT_AUTH` | `10/minute` | General auth endpoints | | `RATE_LIMIT_AUTH_STRICT` | `5/minute` | Login, registration, and password reset | Tighten for public-facing deployments. ## Password reset Users who forget their password request a reset code via `observal auth reset-password --email ` or the web UI **Forgot password?** link. The server logs a 6-character code to its console: ``` WARNING - PASSWORD RESET CODE for alice@example.com: A7X9B2 (expires in 15 minutes) ``` An operator reads the log and passes the code to the user out-of-band (Slack, phone). This is deliberate: no email infrastructure needed for the default flow. If you want emailed reset codes, implement an email transport in the server. ## Operational controls Observal includes: * **Audit logging**: every privileged action lands in ClickHouse's `audit_log` * **SSO-only mode** (`deployment.sso_only=true`) See `docs/self-hosting/sso-cli.md` for SSO CLI commands. ## Next → [Session tracking and reconciliation](/core-concepts/session-tracking) # Upgrades Safe upgrade flow for the Observal server stack. ## Quick upgrade (recommended) If you installed Observal via `observal server start` (the embedded stack), the CLI handles upgrades automatically: ```bash observal server upgrade --dry-run --output json observal server upgrade --version 0.9.0 --force --output json ``` This pulls new Docker images, backs up PostgreSQL, recreates containers, and runs health checks. If the health check fails, it requests the previous image version again. Local shell and Docker access authorize the operation; the command does not require a reachable API or API role. See [`observal server upgrade`](/cli-reference/server#observal-server-upgrade) for full details. ### Server-package upgrades If you installed with `install-server.sh`, rerun the installer for the target release. Setup detects the existing `.env`. Choose the default **No** response when asked to replace configuration so custom values and existing direct credentials remain unchanged. The upgrade records the prior bind address when an older install has no `OBSERVAL_BIND_ADDRESS` setting. Back up `.env`, `secrets/`, and the `apidata` and `pgdata` volumes first. Choosing **Yes** intentionally rebuilds the general configuration from the new template; core application, PostgreSQL, ClickHouse, Grafana, and demo credentials are preserved or migrated into restricted files, but unrelated custom environment entries must be reapplied. ## Before a manual upgrade 1. **Back up `pgdata`** and **`apidata`**. See [Backup and restore](/self-hosting/backup-and-restore). Backing up `chdata` is nice-to-have; losing telemetry is painful but not catastrophic. 2. **Read the** [**CHANGELOG**](https://github.com/Observal/Observal/blob/main/CHANGELOG.md) for the releases you're jumping across. Note any breaking changes. 3. **Pin the version you're upgrading to**: don't `git pull main` blindly. Check out a release tag or a known-good commit. ## Standard upgrade ```bash cd Observal # Fetch and check out the target version git fetch --tags git checkout v0.9.1 # Rebuild images docker compose -f docker/docker-compose.yml pull docker compose -f docker/docker-compose.yml up --build -d # Verify docker compose -f docker/docker-compose.yml ps curl http://localhost/health ``` The init container applies pending Postgres Alembic migrations and ClickHouse SQL migrations before the API starts. Watch the init logs for migration output: ```bash docker logs -f observal-init # Running database migrations... # Running ClickHouse migrations... ``` ## Zero-downtime upgrade (small teams) If you run a single instance and have a \~30-second maintenance window: 1. Back up `pgdata`, `apidata`, `chdata`. 2. Stop the API and worker: `docker compose stop observal-api observal-worker`. 3. Apply migrations out of band with `alembic upgrade head` and `python -m services.clickhouse.migrations` from `observal-server`, or run the init container once. 4. Pull/rebuild new images: `docker compose pull && docker compose build observal-api observal-worker`. 5. Start: `docker compose up -d`. 6. Smoke test: `observal auth status --output json && observal ops telemetry status --output json`. Web UI, Postgres, ClickHouse, Redis stay up throughout. Users see a brief API outage (\~15–30 s). ## Zero-downtime at scale For blue/green upgrades on large deployments: 1. Run a second stack (`docker-compose.yml` with different project name and host ports) behind a reverse proxy. 2. Apply migrations before traffic cutover. Alembic handles Postgres and `services.clickhouse.migrations` handles ClickHouse. Additive migrations should work with API N-1 and N during the rollout. 3. Bring up the green stack pointing at the same `pgdata` / `chdata` / `apidata` volumes. 4. Flip the reverse proxy to green. 5. Decommission blue. If a migration is **not** additive (rare, but happens: column drops, type changes), it gets called out in the CHANGELOG and requires a brief outage. Plan the window. ## Rolling back If the new version breaks: 1. `docker compose -f docker/docker-compose.yml down` 2. `git checkout ` 3. `docker compose -f docker/docker-compose.yml up --build -d` **The catch:** if the failing version already applied a migration, downgrading to the previous API version may leave you running against a schema it doesn't know about. Options: * **If the migration is additive** (most are): the previous version works fine against the newer schema. * **If the migration is destructive**: restore from the pre-upgrade `pgdata` backup. This is why the backup is step 1. ## CLI upgrades CLI upgrades are independent of server upgrades. Users: ```bash observal self upgrade ``` The CLI speaks a stable contract with the server. A newer CLI works against an older server and vice versa, within a release or two. On the first command after upgrading from a release that generated `observal-shim` entries, the CLI restores the original MCP command in known home and current-project config files. It writes a `.pre-unshim.bak` copy before each change. A malformed wrapped entry stops the command with a visible migration error rather than guessing at the original executable. ## Zero-downtime for the web UI The web UI is a static Vite build and restarts instantly. Users see a brief reload if they are on the page during the deploy. No special handling required. ## Next → [Backup and restore](/self-hosting/backup-and-restore) # Backup and restore What to back up, how often, and how to restore. Observal has three persistent stores with very different loss profiles. ## What matters, in order | Priority | Volume | Contents | Loss impact | | ------------- | ------------- | ------------------------------------------------------ | --------------------------------------------------------------------- | | **Critical** | `apidata` | JWT signing keys | Every session invalidated. No recovery without the keys. | | **Critical** | `pgdata` | Users, RBAC, registry metadata, agents | All accounts and registry lost. | | **Important** | `chdata` | Session events, aggregates, audit, and security events | All telemetry lost. Accounts and registry survive. | | **Low** | `grafanadata` | Custom Grafana dashboards | Custom dashboards lost; provisioned defaults come back automatically. | | **Low** | `redisdata` | Job queue state | In-flight jobs lost; they re-queue on next worker restart. | **Always back up `apidata` and `pgdata` together.** Backing up one without the other leaves you in a broken state after restore. ## Backup cadence | Cadence | What | Retention | | -------------------- | ------------------- | ------------------------------------------ | | Daily | `pgdata`, `apidata` | 30 days | | Weekly | `chdata` | 12 weeks | | Before every upgrade | All three | Keep until the upgrade is confirmed stable | ## Postgres backup Use `pg_dump` inside the running container: ```bash docker compose -f docker/docker-compose.yml exec -T observal-db \ pg_dump -U postgres observal | gzip > observal-pg-$(date +%Y%m%d).sql.gz ``` Restore into a fresh DB: ```bash docker compose -f docker/docker-compose.yml down -v docker compose -f docker/docker-compose.yml up -d observal-db zcat observal-pg-20260421.sql.gz | \ docker compose -f docker/docker-compose.yml exec -T observal-db \ psql -U postgres observal docker compose -f docker/docker-compose.yml up -d ``` ## JWT key backup (`apidata` volume) The API container mounts the `apidata` volume at `/data` and stores keys at `/data/keys`. Tar it out: ```bash docker compose -f docker/docker-compose.yml exec -T observal-api \ tar czf - -C /data keys > observal-keys-$(date +%Y%m%d).tar.gz ``` Restore: ```bash # With the API stopped docker compose -f docker/docker-compose.yml stop observal-api docker run --rm \ -v observal_apidata:/data \ -v "$(pwd)":/backup \ alpine sh -c "cd /data && tar xzf /backup/observal-keys-20260421.tar.gz" docker compose -f docker/docker-compose.yml start observal-api ``` ## ClickHouse backup ### Option A - volume snapshot (simplest) ```bash docker compose -f docker/docker-compose.yml stop observal-clickhouse docker run --rm -v observal_chdata:/data -v "$(pwd)":/backup \ alpine tar czf /backup/observal-ch-$(date +%Y%m%d).tar.gz -C /data . docker compose -f docker/docker-compose.yml start observal-clickhouse ``` Downtime: however long the tar takes (a minute to tens of minutes depending on size). ### Option B - ClickHouse native `BACKUP` (no downtime) ```bash docker compose -f docker/docker-compose.yml exec observal-clickhouse \ clickhouse-client --query "BACKUP DATABASE observal TO Disk('backups', 'observal-$(date +%Y%m%d).zip')" ``` Requires configuring a backup disk in ClickHouse config; see [ClickHouse docs](https://clickhouse.com/docs/en/operations/backup). Restore: ```bash docker compose -f docker/docker-compose.yml exec observal-clickhouse \ clickhouse-client --query "RESTORE DATABASE observal FROM Disk('backups', 'observal-20260421.zip')" ``` ## Restore order If you're restoring from backup after a catastrophic failure: 1. Stop the whole stack: `docker compose down`. 2. Restore `apidata` (JWT keys) first. 3. Restore `pgdata` (Postgres). 4. Restore `chdata` (ClickHouse). 5. Bring up the stack: `docker compose up -d`. 6. Smoke test: `observal auth login`, `observal auth status`. Skipping step 2 works but every user has to re-login. ## Verifying a backup Test restores in a staging environment at least quarterly. Untested backups are guesses. Smoke test after restore: ```bash observal auth login observal auth whoami # you should be your pre-backup user observal agent list # registry should be intact observal ops traces --limit 5 # traces up to the backup timestamp should be visible ``` ## Automated backup A minimal cron setup (on the Docker host): ```cron # Daily at 03:00 - Postgres + JWT keys 0 3 * * * cd /opt/Observal && \ docker compose -f docker/docker-compose.yml exec -T observal-db \ pg_dump -U postgres observal | gzip > /backups/pg-$(date +\%Y\%m\%d).sql.gz 0 3 * * * cd /opt/Observal && \ docker compose -f docker/docker-compose.yml exec -T observal-api \ tar czf - -C /data keys > /backups/keys-$(date +\%Y\%m\%d).tar.gz # Weekly Sunday at 04:00 - ClickHouse 0 4 * * 0 cd /opt/Observal && \ docker compose -f docker/docker-compose.yml stop observal-clickhouse && \ docker run --rm -v observal_chdata:/data -v /backups:/backup alpine \ tar czf /backup/ch-$(date +\%Y\%m\%d).tar.gz -C /data . && \ docker compose -f docker/docker-compose.yml start observal-clickhouse ``` Ship the `/backups` directory offsite (S3, B2, rsync to another host). ## Next → [Troubleshooting](/self-hosting/troubleshooting) # Troubleshooting Common failure modes and their fixes. If none of these match, open a [GitHub Discussion](https://github.com/Observal/Observal/discussions) with the output of `observal auth status` and relevant logs from `docker compose logs`. ## Install and CLI ### `"Connection failed. Is the server running?"` The CLI cannot reach the API. Check: ```bash docker compose -f docker/docker-compose.yml ps # API status curl http://localhost/health # API health observal config show # is server_url right? ``` If `server_url` is wrong: ```bash observal config set server_url http://localhost observal auth login ``` ### `"System already initialized"` when logging in The server already has users, so bootstrap is disabled. Use `observal auth login` with an email + password or an API key, not a fresh bootstrap flow. ## Docker and networking ### `port is already allocated` Another process is on one of Observal's default ports. Remap host ports: ```bash POSTGRES_HOST_PORT=5433 REDIS_HOST_PORT=6380 \ docker compose -f docker/docker-compose.yml up --build -d ``` Full list in [Ports and volumes](/self-hosting/ports-and-volumes). ### Service stuck in `starting` The API depends on Postgres, ClickHouse, and Redis being healthy. Check each: ```bash docker compose -f docker/docker-compose.yml ps docker compose -f docker/docker-compose.yml logs observal-db docker compose -f docker/docker-compose.yml logs observal-clickhouse docker compose -f docker/docker-compose.yml logs observal-redis ``` Common causes: * ClickHouse stuck during initial `CREATE TABLE`. Restart it once the healthcheck passes on other DBs * `CLICKHOUSE_PASSWORD` mismatch between services and API config ### Services restart in a loop Check logs (`docker compose logs -f `). Three frequent causes: * Memory limit too tight. Bump limits in `docker-compose.yml` * Corrupt volume. Wipe and restore from backup * Config error introduced during an upgrade. Roll back ## Auth ### Admin forgot password ```bash observal auth reset-password --email admin@demo.example ``` Then read the reset code from the server log: ```bash docker logs observal-api 2>&1 | grep "PASSWORD RESET CODE" ``` Enter the code when the CLI prompts. ### OAuth login fails with `redirect_uri_mismatch` The IdP doesn't have the right redirect URI registered. Add: ``` {FRONTEND_URL}/api/v1/auth/oauth/callback ``` with `FRONTEND_URL` set to your real external URL (scheme and host must match exactly). ### All users logged out after restart Likely the `apidata` volume was recreated, so the JWT signing keys are new. Restore the `apidata` volume from backup, or accept that all sessions are invalid and everyone has to log in again. ## Telemetry ### Nothing in the dashboard Run through, in order: ```bash # 1. Are sessions arriving at all? observal ops telemetry status # 2. Are session hooks installed for the harness? observal doctor --output json # 3. Is the API reachable from the harness environment? curl http://localhost/health ``` If hooks are missing, run `observal doctor patch --harness `. If sessions still are not arriving, check `~/.observal/telemetry_buffer.db`; growth indicates pending session delivery rather than silent loss. ### ClickHouse not receiving data Check the `CLICKHOUSE_URL` the API is using: ```bash docker compose -f docker/docker-compose.yml exec observal-api \ printenv CLICKHOUSE_URL ``` The source Compose default is `clickhouse://default:clickhouse@observal-clickhouse:8123/observal`. Mismatches typically happen after changing `CLICKHOUSE_PASSWORD` without updating the URL. Server-package installs use `CLICKHOUSE_URL_FILE=/run/secrets/clickhouse_url`, a hashed ClickHouse user configuration, and a separate health-check password file. Confirm the file is mounted without printing it: ```bash docker compose exec observal-api test -r /run/secrets/clickhouse_url docker compose exec observal-clickhouse test -r /run/secrets/clickhouse_password ``` Verify ClickHouse itself: ```bash docker compose -f docker/docker-compose.yml exec observal-clickhouse \ clickhouse-client --query "SELECT count() FROM observal.session_events" ``` ## Web UI ### Blank white page Frontend is still building. Check: ```bash docker compose -f docker/docker-compose.yml logs -f observal-web ``` For local dev (running Next.js outside Docker), verify `NEXT_PUBLIC_API_URL` in `web/.env.local` matches your backend. ### Login redirects back to login immediately Browser cookies aren't being set. Usually one of: * `FRONTEND_URL` doesn't match the URL you're hitting. * `CORS_ALLOWED_ORIGINS` doesn't include your frontend origin. * You're on HTTP behind a proxy that's setting `secure` cookies. Terminate TLS at your proxy and keep `FRONTEND_URL=https://...`. ## Where to get more help * Logs: `docker compose -f docker/docker-compose.yml logs -f` * Health: `curl http://localhost/health` * Status: `observal auth status` * Community: [GitHub Discussions](https://github.com/Observal/Observal/discussions) * Bugs: [GitHub Issues](https://github.com/Observal/Observal/issues). Please use Discussions for questions, Issues only for confirmed bugs # Copilot Copilot is a first-class Observal harness integration. Observal can install GitHub Copilot agents, configure MCP servers, expose skills, and bridge telemetry hooks. *** ## Overview Copilot agent profiles are Markdown files stored in `.github/agents/`. Copilot currently supports project installation scope only. MCP servers, hooks, skills, and agent profiles are managed through the project configuration. *** ## Supported capabilities | Capability | Support | | --------------- | ------------------------------------- | | Agent profiles | Project scope | | Hook bridge | Supported | | MCP servers | `.vscode/mcp.json` | | Skills | `.github/skills/{name}/SKILL.md` | | Session parsing | Built-in `copilot-cli` session parser | | Default scope | Project | *** ## Setup ### 1. Install the Observal CLI ```bash uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash observal auth login ``` This writes credentials to `~/.observal/config.json`. ### 3. Pull an agent into Copilot ```bash observal agent pull --harness copilot ``` By default, the agent is written to: ``` .github/agents/{name}.agent.md ``` *** ## Config paths | Purpose | Project scope | | ------------- | -------------------------------- | | Agent profile | `.github/agents/{name}.agent.md` | | MCP config | `.vscode/mcp.json` | | Skills | `.github/skills/{name}/SKILL.md` | | Hook config | `.github/hooks/{name}.json` | Copilot MCP configuration uses the `servers` key. *** ## Hook spec ### Event map | Observal event | Copilot event | | ------------------ | ------------------ | | `SessionStart` | `SessionStart` | | `UserPromptSubmit` | `UserPromptSubmit` | | `PreToolUse` | `PreToolUse` | | `PostToolUse` | `PostToolUse` | | `Stop` | `Stop` | *** ## Session parsing and delivery Copilot uses the built-in `copilot-cli` server parser while retaining separate source adapters for VS Code Copilot and Copilot CLI. VS Code hook events are durably materialized as JSONL-compatible source records; Copilot CLI discovers its native `events.jsonl` sessions. Both adapters route through the shared spool-first delivery engine. Hook execution writes to the durable outbox and defers network work, retries keep stable source indexes, and cursors advance only after contiguous server acknowledgement. Recovery uses the authenticated server checkpoint. Finalization audits the complete source hash and rewinds/replays a mismatched range. *** ## Caveats **Only project scope is supported.** Agents are installed into `.github/agents/`. **Configuration is split across project directories.** MCP servers are stored in `.vscode/mcp.json`, while hooks and skills live under `.github/`. **Rules are not supported.** Copilot supports agent profiles, MCP servers, skills, and hooks, but does not expose a rules configuration. # Cursor Cursor is a first-class Observal harness integration. Observal can install Cursor agents, configure MCP servers, and collect Cursor session telemetry. *** ## Overview Cursor agent profiles are Markdown files. Project agents live in `.cursor/agents/`. User agents live in `~/.cursor/agents/`. Cursor supports project and user installation scopes. MCP servers, hooks, and agent profiles are managed under the `.cursor` configuration directory. *** ## Supported capabilities | Capability | Support | | --------------- | ------------------------------------------- | | Agent profiles | Project and user scope | | Hook bridge | Supported | | MCP servers | `.cursor/mcp.json` and `~/.cursor/mcp.json` | | Session parsing | Built-in Cursor session parser | | Default scope | Project | *** ## Setup ### 1. Install the Observal CLI ```bash uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash observal auth login ``` This writes credentials to `~/.observal/config.json`. ### 3. Pull an agent into Cursor ```bash observal agent pull --harness cursor ``` Cursor's default scope is project scope. By default, the agent is written to `.cursor/agents/{name}.md`. To install into your user configuration: ```bash observal agent pull --harness cursor --scope user ``` User agents are written to `~/.cursor/agents/{name}.md`. *** ## Config paths | Purpose | Project scope | User scope | | ------------- | -------------------------------- | ---------------------------------- | | Agent profile | `.cursor/agents/{name}.md` | `~/.cursor/agents/{name}.md` | | MCP config | `.cursor/mcp.json` | `~/.cursor/mcp.json` | | Skills | `.cursor/skills/{name}/SKILL.md` | `~/.cursor/skills/{name}/SKILL.md` | | Hook config | `.cursor/hooks.json` | `~/.cursor/hooks.json` | Cursor MCP configs use the `mcpServers` key. *** ## Hook spec ### Event map | Observal event | Cursor event | | ------------------ | -------------------- | | `PreToolUse` | `preToolUse` | | `PostToolUse` | `postToolUse` | | `Stop` | `sessionEnd` | | `SessionStart` | `sessionStart` | | `UserPromptSubmit` | `beforeSubmitPrompt` | | `SubagentStop` | `subagentStop` | *** ## Session delivery and parsing Cursor keeps its built-in `cursor` parser. Its adapter resolves the transcript path supplied by Cursor, with project-directory discovery as a fallback, and discovers separately stored subagent transcripts. Cursor hook timeouts require a split delivery path: the hook synchronously writes complete transcript records and Stop-event token usage into the durable SQLite outbox, then a detached shared worker performs network delivery. The local byte/line cursor is not advanced by the hook or worker launch; it advances only after the server acknowledges a contiguous checkpoint. A delayed finalizer captures records written after Cursor's Stop event, restores stale local state from the server checkpoint, and performs the final hash audit. Audit mismatches rewind and replay the affected range. *** ## Caveats **Default scope is project.** `observal agent pull --harness cursor` writes to `.cursor/agents/` unless `--scope user` is specified. **Configuration lives under `.cursor`.** Agent profiles, MCP configuration, skills, and hooks are stored in the `.cursor` directory for project installs and under `~/.cursor` for user installs. **Auto model sentinel configuration is not available.** # Goose [Goose](https://goose-docs.ai/) is a first-class Observal harness integration. Observal can install Goose custom agents, configure MCP extensions, expose skills, install a Goose hook plugin, and collect Goose session telemetry. Supported Goose versions: **1.10.0 and later** (the release that moved session storage to SQLite), verified against **goose 1.45.0**. Goose CLI and Goose Desktop share the same configuration, skills, agents, plugins, and session database, so a single install covers both. *** ## Overview Goose custom agents are Markdown files with YAML frontmatter. Project agents live in `.agents/agents/`, user agents in `~/.agents/agents/`. MCP servers are Goose *extensions*. Goose reads them from a single user-level `config.yaml`, so Observal merges its extensions into that file and leaves every other key — providers, models, global settings — untouched. Session telemetry uses a Goose hook plugin. `observal doctor patch --harness goose` writes `~/.agents/plugins/observal/`, containing a `plugin.json` manifest and a `hooks/hooks.json` file. The hooks run `observal_cli.hooks.session_push --harness goose`, which reads Goose's SQLite session store and pushes new records through Observal's shared acknowledged delivery pipeline. *** ## Supported capabilities | Capability | Support | | --------------- | ----------------------------------------------------------------------- | | Agent profiles | Project and user scope | | Hook bridge | Goose plugin (`hooks/hooks.json`) | | Hook events | `SessionStart`, `UserPromptSubmit`, `Stop`, `SessionEnd` | | MCP servers | `extensions` in `~/.config/goose/config.yaml` | | MCP transports | `stdio` and `streamable_http` | | Agent prompt | Registry prompts are written into the generated Goose agent file | | Skills | `.agents/skills/{name}/SKILL.md` and `~/.agents/skills/{name}/SKILL.md` | | Session parsing | SQLite session store mirrored to JSONL | | Telemetry | Hook push plus `observal reconcile` | | Model selection | Registry-backed Goose model catalog | *** ## Setup ### 1. Install the Observal CLI ```bash uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash observal auth login ``` ### 3. Pull an agent into Goose ```bash observal agent pull --harness goose ``` Goose's default scope is user scope, so the agent is written to `~/.agents/agents/{name}.md`. To install into the current project: ```bash observal agent pull --harness goose --scope project ``` MCP extensions are always written to `~/.config/goose/config.yaml`, because that is the only file Goose reads extensions from. ### 4. Install or refresh the Goose hook plugin ```bash observal doctor patch --harness goose ``` Restart your Goose session afterwards so the plugin is discovered. ### 5. Check what is installed ```bash observal scan --harness goose observal doctor ``` *** ## Config paths | Purpose | Project scope | User scope | | -------------------- | -------------------------------- | ------------------------------------------- | | Agent profile | `.agents/agents/{name}.md` | `~/.agents/agents/{name}.md` | | MCP extensions | — | `~/.config/goose/config.yaml` | | Skill definition | `.agents/skills/{name}/SKILL.md` | `~/.agents/skills/{name}/SKILL.md` | | Hook plugin | `.agents/plugins/observal/` | `~/.agents/plugins/observal/` | | Session database | — | `~/.local/share/goose/sessions/sessions.db` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Observal lockfile | `~/.observal/lockfile.json` | `~/.observal/lockfile.json` | User-scope locations follow Goose's own resolution order: 1. `$GOOSE_PATH_ROOT` when set to an absolute path — config becomes `$GOOSE_PATH_ROOT/config`, data `$GOOSE_PATH_ROOT/data`, and skills, agents and plugins move to `$GOOSE_PATH_ROOT/.agents/`. 2. `$XDG_CONFIG_HOME` / `$XDG_DATA_HOME` when set. 3. `~/.config/goose` and `~/.local/share/goose`. On Windows, Goose reads `%APPDATA%\Block\goose\config\config.yaml` and `%APPDATA%\Block\goose\data\sessions\sessions.db`. Scanning, `doctor`, and session reading resolve those locations automatically. `observal agent pull` still writes the documented Unix paths, so Windows users should move the generated `config.yaml` extensions into the `%APPDATA%` config by hand. *** ## Agent profile format ```markdown --- name: my-agent description: Reviews code for correctness and risk model: gpt-5.5 --- You are a Goose agent with the following specialization... ``` Only `name` is required by Goose. `model` is emitted when the install resolves a model from the Goose model catalog. *** ## Extension format Observal writes MCP servers as Goose extensions: ```yaml extensions: filesystem: type: stdio name: filesystem enabled: true cmd: npx args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] envs: {} env_keys: [] timeout: 300 remote-tools: type: streamable_http name: remote-tools enabled: true uri: https://example.com/mcp headers: {} envs: {} env_keys: [] timeout: 300 ``` Observal writes MCP servers as `stdio` or `streamable_http` extensions and only ever touches the entries it owns. `observal scan` additionally reports legacy `sse` extensions. Goose's `builtin`, `platform` and `frontend` extensions run in-process and `inline_python` carries code rather than a command, so none of them are reported or rewritten. *** ## Hook plugin spec `~/.agents/plugins/observal/hooks/hooks.json`: ```json { "hooks": { "SessionStart": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }], "UserPromptSubmit": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }], "Stop": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }], "SessionEnd": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }] } } ``` Rules deliberately omit `matcher`: Goose treats `matcher` as a regular expression, so a bare `"*"` would be invalid and the rule would be skipped. Goose runs every hook through `sh -c` on all platforms, passes the event JSON on stdin, waits for the command, and enforces the rule's `timeout`. The Observal hook therefore only reads Goose's database and appends to the local outbox; the network upload is handed to a detached worker so a slow or unreachable Observal server can never stall a Goose turn. The hook writes nothing to stdout and never exits `2`, so it can never block a `PreToolUse` tool call or a `Stop` turn boundary. `doctor patch` preserves any non-Observal rules you add to the same plugin. Goose records each discovered plugin under a `plugins` map in `config.yaml` keyed by its absolute path, so the plugin activates itself on the next session. To disable the plugin without removing it, add it to `disabledPlugins` in `~/.config/goose/settings.json`. To remove it entirely: ```bash observal doctor cleanup --harness goose ``` ### Hook payload coverage Verified against goose 1.45.0: | Event | `working_dir` | Notes | | ------------------ | ------------- | --------------------------------------------------- | | `SessionStart` | no | Fires once per session | | `UserPromptSubmit` | no | Carries the prompt in `message` | | `Stop` | yes | Fires at the end of **every turn**, not the session | | `SessionEnd` | no | Emitted by the goose **CLI** only | Because most events omit `working_dir`, Observal reads the session's working directory from `sessions.working_dir` instead, which keeps agent attribution and layer hashing correct on every event. *** ## Session push behavior Goose stores sessions in SQLite rather than JSONL, so the CLI projects each session onto an append-only JSONL mirror at `~/.observal/sessions/goose/.jsonl`. Everything downstream — the durable outbox, acknowledged delivery, checkpoint recovery, and the final SHA-256 audit — is the shared engine used by every other harness. 1. A hook event resolves the Goose session id from the payload. 2. `sessions.db` is opened through a `file:...?mode=ro` URI with a busy timeout, so Observal never writes to, migrates, or checkpoints Goose's database, and never blocks a Goose session that is writing concurrently. 3. Rows newer than the mirror's cursor are appended as JSONL records. The mirror only grows, which keeps previously acknowledged byte offsets valid. 4. The batch is spooled to the local outbox and a detached worker performs the upload, so the hook returns in milliseconds regardless of server health. 5. `SessionEnd` appends a `session_end` boundary carrying the session's token and cost totals, then finalizes delivery with an integrity hash. 6. Delegated (subagent) sessions linked by `parent_session_id` are mirrored and delivered alongside their parent. Mirror records: | Record | Meaning | | ------------------------------ | -------------------------------------------------------------------------- | | `{"type": "session", ...}` | Session metadata: name, working directory, provider, model, parent session | | `{"type": "message", ...}` | One `messages` row: role, content blocks, per-message token usage | | `{"type": "session_end", ...}` | Final boundary with accumulated token and cost totals | Goose content blocks are parsed into trace events: `text`, `thinking`, `redactedThinking`, `toolRequest`, `toolResponse` (including failures), `toolConfirmationRequest`, and `error`. ### Backfilling existing sessions ```bash observal reconcile --harness goose observal reconcile --harness goose --since 720 --dry-run ``` Reconcile reads the same read-only database, mirrors any session updated within the window, and drains it through the outbox. *** ## Caveats **The hook plugin is shared per machine.** It is installed by `doctor patch`, not by each agent pull. **MCP extensions are user scope only.** Goose reads extensions from one `config.yaml`, so `--scope project` still writes extensions to the user config. Agents and skills honour the requested scope. Goose rewrites that same file when it registers plugins, and neither Goose nor Observal preserves YAML comments in it. **Goose Desktop sessions finalize later.** Only the goose CLI emits `SessionEnd`. Desktop sessions are still captured turn by turn through `Stop`; their final integrity audit runs from background recovery or `observal reconcile`. **Legacy JSONL sessions are not imported.** Goose imports pre-1.10 `.jsonl` files into `sessions.db` on upgrade, and Observal reads the database, so those sessions are covered. Leftover `.jsonl` files on disk are ignored. **Rewritten conversations replay.** `goose session --resume --edit` rewrites a session's messages. The mirror is append-only, so the rewritten history is appended rather than replacing what was already delivered. **Late writes land on the next event.** If Goose persists a message after the `SessionEnd` hook has read the database, that row is delivered by the next hook event or by `observal reconcile`. # Kiro Kiro is a first-class Observal harness integration. Observal can install Kiro agents, configure MCP servers, add hooks, expose skills, and collect Kiro session telemetry. *** ## Overview Kiro agent profiles are JSON files. Project agents live in `.kiro/agents/`. User agents live in `~/.kiro/agents/`. When Observal installs a Kiro agent, it writes telemetry hooks into a standalone hooks file — `~/.kiro/hooks/observal.json` (user scope) or `.kiro/hooks/observal.json` (project scope) — in the v1 schema, using the `UserPromptSubmit` and `Stop` triggers. Both hooks run the shared `observal_cli.hooks.session_push --harness kiro` entry point. Kiro IDE 1.0 and CLI 3.0 read the standalone file; the IDE does not fire inline agent hooks. Once installed, shared hooks collect **all** Kiro sessions in that scope, including ordinary conversations using MCPs or skills. Agent attribution is separate: sessions without a verified registry agent remain unattributed. Inline hooks do **not** hide an agent from the IDE agent picker. What does is `allowedTools` or `toolsSettings` without a `permissions` block: Kiro IDE 1.x `ProfileLoader` rejects such a profile outright (`reasonCode cli_only_agent`). Observal no longer emits either field and strips them on pull. ### Legacy Kiro CLI 2.x Kiro CLI 2.x only understands inline agent hooks. When Observal detects CLI 2.x, it additionally writes legacy inline `userPromptSubmit`/`stop` hooks into the agent JSON, even when the IDE is installed. Ordinary CLI 2.x sessions do not fire these agent-specific hooks; use `observal reconcile --harness kiro` to backfill them. Detection can be overridden with `OBSERVAL_KIRO_CLI_VERSION` and `OBSERVAL_KIRO_IDE` (`1`/`0`). The hook reads Kiro CLI JSONL files from `~/.kiro/sessions/cli/` and IDE transcripts from workspace session directories. It sends new records since the last push. Reconciliation backfills missed CLI and IDE sessions, including separately stored delegated sub-executions. *** ## Supported capabilities | Capability | Support | | --------------- | --------------------------------------------------------------------------------------------------------------------------- | | Agent profiles | Project and user scope | | Hook bridge | `UserPromptSubmit` and `Stop` in `.kiro/hooks/observal.json` | | Custom hooks | `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `Stop`, `PostFileSave`, `PostFileCreate`, `PostFileDelete` | | Legacy CLI 2.x | Inline agent hooks, only when no Kiro IDE is installed | | MCP servers | `.kiro/settings/mcp.json` and `~/.kiro/settings/mcp.json` | | Agent prompt | Registry prompts are embedded in the generated Kiro agent profile | | Guidance files | Scanned from steering files and `AGENTS.md`, not overwritten | | Skills | `.kiro/skills/{name}/SKILL.md` and `~/.kiro/skills/{name}/SKILL.md` | | Session parsing | Kiro JSONL parser | | Telemetry | Kiro session transcripts delivered through hooks and reconciliation | | Model selection | Registry-backed Kiro model catalog | *** ## Setup ### 1. Install the Observal CLI ```bash uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash observal auth login ``` This writes credentials to `~/.observal/config.json`. ### 3. Pull an agent into Kiro ```bash observal agent pull --harness kiro ``` Kiro's default scope is user scope. By default, the agent is written to `~/.kiro/agents/{name}.json`. To install into the current project: ```bash observal agent pull --harness kiro --scope project ``` Project agents are written to `.kiro/agents/{name}.json`. ### 4. Refresh Kiro hooks Pull the agent again to refresh its Observal hook commands. `doctor patch` does not install generic Kiro hooks: pulling an agent installs shared hooks for its scope. These hooks do not carry that agent's UUID and capture ordinary sessions too. CLI 2.x still needs per-agent inline hooks. *** ## Config paths | Purpose | Project scope | User scope | | -------------------- | ----------------------------------------- | ----------------------------------------- | | Agent profile | `.kiro/agents/{name}.json` | `~/.kiro/agents/{name}.json` | | Guidance files | `.kiro/steering/*.md`, `AGENTS.md` | `~/.kiro/steering/*.md` | | MCP config | `.kiro/settings/mcp.json` | `~/.kiro/settings/mcp.json` | | Skill definition | `.kiro/skills/{name}/SKILL.md` | `~/.kiro/skills/{name}/SKILL.md` | | Hook config | Embedded in `.kiro/agents/{name}.json` | Embedded in `~/.kiro/agents/{name}.json` | | Custom hook scripts | `.kiro/hooks/` | `~/.kiro/hooks/` | | Session JSONL | `~/.kiro/sessions/cli/{session_id}.jsonl` | `~/.kiro/sessions/cli/{session_id}.jsonl` | | Credit metadata | `~/.kiro/sessions/cli/{session_id}.json` | `~/.kiro/sessions/cli/{session_id}.json` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Last session cache | `~/.observal/.kiro-session` | `~/.observal/.kiro-session` | Kiro MCP configs use the `mcpServers` key. *** ## Hook spec Observal writes the telemetry hooks to the standalone v1 hooks file — `~/.kiro/hooks/observal.json` (user scope) or `.kiro/hooks/observal.json` (project scope): ```json { "version": "v1", "hooks": [ { "name": "observal-session-push-userpromptsubmit", "trigger": "UserPromptSubmit", "action": { "type": "command", "command": "python -m observal_cli.hooks.session_push --harness kiro" } }, { "name": "observal-session-push-stop", "trigger": "Stop", "action": { "type": "command", "command": "python -m observal_cli.hooks.session_push --harness kiro" } } ] } ``` The command carries no `OBSERVAL_AGENT_ID`. One file per scope serves every agent, so an id baked into it would attribute every session to whichever agent was pulled last. Attribution comes from session metadata instead. On non-Windows platforms, generated server config may use `python3` instead of `python`. During `observal agent pull`, the CLI rewrites Observal hook commands to use the active Python interpreter. ### Legacy CLI 2.x inline hooks Kiro CLI 2.x predates the standalone file and reads hooks only from the agent JSON. On a machine whose `kiro-cli` reports major version 2, Observal *also* writes inline `userPromptSubmit`/`stop` hooks into each agent profile, carrying `OBSERVAL_AGENT_ID` since an agent profile belongs to exactly one agent: ```json { "hooks": { "userPromptSubmit": [ { "command": "OBSERVAL_AGENT_ID= python -m observal_cli.hooks.session_push --harness kiro" } ] } } ``` Both formats are written on such machines. Having the IDE installed is not a reason to withhold the inline copy: the two surfaces coexist, and the IDE loads an agent carrying inline hooks and simply never fires them. ### Attribution The hooks file is shared by every agent, so the hook command cannot identify one. Attribution is resolved from the session itself: 1. A session started as an agent — picked from the IDE's agent dropdown — records that agent in `session_start.agentType`, and the agent's own profile hooks fire with an agent-scoped `hookId`. Either names the agent. 2. A session that delegates records `sub_agent_start` with `subAgentName`, and the delegated agent's work is written to a separate child transcript. The parent conversation is collected but is **not** attributed to the delegate. 3. CLI sessions carry the active agent in the session metadata written beside the transcript. 4. The CLI selects the active server URL under `registries` in `~/.observal/lockfile.json`, then looks up that name under that registry's `kiro` harness. 5. A verified agent receives its lockfile id and version. Ordinary sessions and unknown agents are still collected, with no agent id rather than a guessed one. Legacy CLI 2.x inline hooks may carry `OBSERVAL_AGENT_ID`. ### Event map | Observal event | Kiro event | | ------------------ | ------------------ | | `SessionStart` | `agentSpawn` | | `UserPromptSubmit` | `userPromptSubmit` | | `PreToolUse` | `preToolUse` | | `PostToolUse` | `postToolUse` | | `Stop` | `stop` | `preToolUse` and `postToolUse` hooks can include a `matcher`. Observal uses `*` when no matcher is set. *** ## Session push behavior Kiro uses the shared acknowledged session delivery engine: 1. Resolve the Kiro session ID from the hook payload or `~/.observal/.kiro-session`. 2. Find `~/.kiro/sessions/cli/{session_id}.jsonl` through the Kiro adapter. 3. Read complete records after the acknowledged byte/line cursor. 4. Persist new batches to `~/.observal/telemetry_buffer.db` before network delivery. 5. Retry batches idempotently until the server returns a contiguous checkpoint covering them. 6. Advance the local cursor only to that acknowledged checkpoint. 7. Recover missing or corrupt local state from the authenticated server checkpoint. 8. On finalization, compare the SHA-256 audit manifest and replay any affected range. On `stop`, a delayed stable-file pass captures late records and finalizes the cursor. The adapter also reads `~/.kiro/sessions/cli/{session_id}.json` and durably sends Kiro credit usage, including when no transcript lines were added by the final hook. *** ## Agent profile format Observal generates Markdown agent profiles like this: ```markdown --- name: my-agent model: claude-sonnet-4 --- You are a Kiro agent with the following specialization... ``` The `model` field is present when a model is resolved for the agent. *** ## Skill file format Kiro skills live at: | Scope | Path | | ------- | -------------------------------- | | Project | `.kiro/skills/{name}/SKILL.md` | | User | `~/.kiro/skills/{name}/SKILL.md` | Example: ```markdown --- description: "Runs the project test suite" task_type: testing --- # Run Tests Run `pytest -q` from the project root. ``` *** ## Caveats **Guidance files are scan-only.** Observal layers Kiro steering files and `AGENTS.md` as context, but does not overwrite them during pull. **Hooks are per scope, not per agent.** Pulling an agent refreshes the shared `observal.json` hooks file for that scope, which then serves every Kiro agent there — so the file exists even when no agent is locked. Pull the agent again to replace an older Kiro-specific push command with the shared acknowledged exporter. On legacy CLI 2.x, inline per-agent hooks carrying `OBSERVAL_AGENT_ID` are written in addition. **Default scope is user.** `observal agent pull --harness kiro` writes to `~/.kiro/agents/` unless `--scope project` is set. **No Claude Code subagent layout.** Kiro reads `~/.kiro/sessions/cli/{session_id}.jsonl`. It does not scan Claude Code's `subagents/` directory. **MCP config is Kiro-specific.** Kiro uses `.kiro/settings/mcp.json` and `~/.kiro/settings/mcp.json`, not Claude Code MCP paths. # OpenCode OpenCode is a first-class Observal harness integration. Observal can install OpenCode agents, configure MCP servers, expose skills, install hook plugins, and collect OpenCode session telemetry. *** ## Overview OpenCode agent profiles are Markdown files. Project agents live in `.opencode/agents/`. User agents live in `~/.config/opencode/agents/`. OpenCode session telemetry uses an in-process TypeScript plugin. The plugin is installed by `observal auth login` or `observal doctor patch`. Agent pulls write agent profiles and update the Observal lockfile, but they do not embed the telemetry plugin in each agent file. The plugin listens to OpenCode session and message events, reads message data through the OpenCode client API, converts new messages into Observal session lines, and sends them to Observal. *** ## Supported capabilities | Capability | Support | | --------------- | ---------------------------------------------------------------------------------- | | Agent profiles | Project and user scope | | Hook bridge | OpenCode plugin | | Plugin events | `session.created`, `session.idle`, `message.updated` | | MCP servers | `opencode.json` and `~/.config/opencode/opencode.json` | | Agent prompt | Registry prompts are written into the generated OpenCode agent profile | | Skills | `.opencode/skills/{name}/SKILL.md` and `~/.config/opencode/skills/{name}/SKILL.md` | | Session parsing | OpenCode plugin exports JSONL-compatible session lines | | Telemetry | OpenCode session transcripts delivered through the plugin and reconciliation | | Model selection | Registry-backed OpenCode model catalog | *** ## Setup ### 1. Install the Observal CLI ```bash uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash observal auth login ``` This writes credentials to `~/.observal/config.json`. If OpenCode is detected, login can install the Observal plugin. ### 3. Pull an agent into OpenCode ```bash observal agent pull --harness opencode ``` OpenCode's default scope is user scope. By default, the agent is written to `~/.config/opencode/agents/{name}.md`. To install into the current project: ```bash observal agent pull --harness opencode --scope project ``` Project agents are written to `.opencode/agents/{name}.md`. ### 4. Install or refresh the OpenCode plugin ```bash observal doctor patch --harness opencode ``` This installs or updates `observal-plugin.ts` when the plugin is missing, stale, or different from the bundled source. *** ## Config paths | Purpose | Project scope | User scope | | -------------------- | -------------------------------------- | ----------------------------------------------- | | Agent profile | `.opencode/agents/{name}.md` | `~/.config/opencode/agents/{name}.md` | | MCP config | `opencode.json` | `~/.config/opencode/opencode.json` | | Skill definition | `.opencode/skills/{name}/SKILL.md` | `~/.config/opencode/skills/{name}/SKILL.md` | | Plugin hooks | `.opencode/plugins/{name}.ts` | `~/.config/opencode/plugins/{name}.ts` | | Observal plugin | `.opencode/plugins/observal-plugin.ts` | `~/.config/opencode/plugins/observal-plugin.ts` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Observal lockfile | `~/.observal/lockfile.json` | `~/.observal/lockfile.json` | OpenCode MCP configs use the `mcp` key. *** ## Plugin spec Observal installs a TypeScript plugin named `observal-plugin.ts`. The plugin subscribes to OpenCode runtime events: | OpenCode event | Observal use | | ----------------------------- | ----------------------------------------------------------- | | `session.created` | Capture the active OpenCode agent name for the session | | `message.updated` | Mark the session as having new data to push | | `session.idle` | Fetch new messages and send them to Observal | | `session.updated` | Refresh active agent tracking when emitted by OpenCode | | `session.next.agent.switched` | Refresh active agent tracking when the active agent changes | The bundled spec version is checked by `doctor patch`, along with the plugin content hash, so stale local copies are replaced. *** ## Attribution OpenCode exposes the active agent name in session events, not an Observal UUID. The Observal lockfile maps that OpenCode agent name back to the registry agent id and installed version. 1. `observal agent pull` writes the OpenCode agent profile and records the agent name, id, version, scope, and project directory in `~/.observal/lockfile.json`. 2. The OpenCode plugin receives `session.created` or agent switch events and reads the active OpenCode agent name. 3. Built-in OpenCode agents such as `build`, `plan`, `general`, `explore`, `scout`, `compaction`, `title`, and `summary` are ignored. 4. The plugin selects the configured server URL under `registries` in `~/.observal/lockfile.json`, then searches that registry's `opencode` harness for an entry whose `name` or `id` matches the active agent. 5. If multiple entries match, a project-scoped entry for the current directory wins, then a user-scoped entry, then the first match. 6. The session payload is sent with the resolved `agent_id` and `agent_version`. 7. If no lockfile match exists, the plugin skips attribution for that session instead of guessing. *** ## Session push behavior The OpenCode plugin implements the same acknowledged delivery contract as Python harnesses: 1. Load `~/.observal/config.json` and resolve the active agent through `~/.observal/lockfile.json`. 2. On `session.idle`, recover missing, stale, or quarantined local state from the server's contiguous checkpoint. 3. Retry any durable pending batch before reading new messages. 4. Fetch messages from `client.session.messages` and convert each message into one stable JSONL-compatible source record. 5. Atomically persist the pending indexed batch under `~/.observal/opencode_session_outbox/` before network delivery. 6. Retry the same batch idempotently until the server returns a contiguous acknowledgement covering it. 7. Advance the persisted source line only after that acknowledgement; a crash after server commit safely retries the batch. 8. On finalization, compare a SHA-256 audit manifest and replay from the server-requested repair range when needed. Acknowledged state is retained for seven days to avoid unnecessary replay. Capacity errors fail closed without discarding pending records. Corrupt state is quarantined and rebuilt from the authenticated server checkpoint plus the OpenCode message source. *** ## Agent profile format OpenCode agents are Markdown files. Observal writes the agent prompt into the profile selected by scope: ```markdown --- name: my-agent --- You are an OpenCode agent with the following specialization... ``` *** ## Caveats **The plugin is shared per OpenCode install.** It is installed by auth login or `doctor patch`, not by each agent pull. **Attribution depends on the lockfile.** If an agent profile is copied by hand without running `observal agent pull`, the plugin may see the OpenCode agent name but have no Observal id or version to send. **Built-in OpenCode agents are ignored.** Sessions for built-in agents are not attributed to registry agents. **MCP config is OpenCode-specific.** OpenCode uses `opencode.json` and the `mcp` key, not Kiro or Claude Code MCP paths. # Pi Pi is a first-class Observal harness integration. Observal can install Pi agent profiles, configure MCP servers, expose skills, install the telemetry extension, and collect Pi session telemetry. *** ## Overview Pi is harness-centric: the whole of Pi is one agent. There is no per-agent profile file the way Kiro or OpenCode have one. Instead, Observal writes the agent's rules into `AGENTS.md`, which becomes Pi's system prompt, and writes MCP servers and skills next to it. To let several registry agents coexist, a user-scope `observal agent pull` writes each agent into its own profile directory under `~/.pi/agent/agents/{agent}/`. The Observal extension's `/agent` command swaps a profile into the live `~/.pi/agent/` location. A project-scope pull writes the rules straight to the project's `AGENTS.md`, which Pi reads directly. Pi session telemetry uses an in-process TypeScript extension, `~/.pi/agent/extensions/observal.ts`. It is installed by `observal doctor patch` and is shared by every agent; agent pulls do not embed telemetry hooks. *** ## Supported capabilities | Capability | Support | | ---------------- | --------------------------------------------------------------------------------------------------------------------------- | | Agent profiles | Project and user scope, as `AGENTS.md` | | Hook bridge | Pi extension (no shell hooks) | | Extension events | `session_start`, `agent_end`, `session_shutdown` | | MCP servers | `.pi/mcp.json` and `~/.pi/agent/mcp.json` | | Agent prompt | Registry rules are written into the generated `AGENTS.md` | | Guidance files | Scanned from `AGENTS.md`, `~/.pi/agent/AGENTS.md`, `.pi/SYSTEM.md`, `.pi/APPEND_SYSTEM.md` | | Skills | `.pi/skills/{name}/SKILL.md` and `~/.pi/agent/skills/{name}/SKILL.md` | | Session parsing | Pi JSONL parser | | Telemetry | Pi session transcripts delivered through the extension; `observal reconcile --harness pi` is accepted but finds no sessions | | Model selection | Registry-backed Pi model catalog (`observal registry models list --harness pi`) | *** ## Setup ### 1. Install the Observal CLI ```bash uv tool install observal-cli # or: pipx install observal-cli ``` Pi 0.74.0 or newer is required by the telemetry extension. ### 2. Authenticate ```bash observal auth login ``` This writes credentials to `~/.observal/config.json`. Unless you pass `--no-setup` or `--output json`, login then installs the bundled Observal skills into every detected harness (Pi counts once `~/.pi/` exists) and runs `observal doctor`, which warns if the Pi telemetry extension is missing or stale. A healthy install prints no Pi warning. ### 3. Pull an agent into Pi ```bash observal agent pull --harness pi ``` Pi's default scope is user scope. The agent is written to `~/.pi/agent/agents/{agent-name}/`. To install into the current project: ```bash observal agent pull --harness pi --scope project ``` Project pulls write the rules to `AGENTS.md` in the project root, which Pi reads directly, and the agent's MCP servers and skills to `.pi/agents/{agent-name}/`. Pi loads project MCP servers from `.pi/mcp.json` and project skills from `.pi/skills/{name}/SKILL.md`, and `/agent` handles only user-scope profiles, so activate them by hand: ```bash cp .pi/agents//mcp.json .pi/mcp.json cp -r .pi/agents//skills/. .pi/skills/ ``` Merge instead of copying if `.pi/mcp.json` already lists other servers. ### 4. Install or refresh the telemetry extension ```bash observal doctor patch --harness pi ``` This writes the bundled extension to `~/.pi/agent/extensions/observal.ts` when it is missing, and refreshes a copy Observal recognises as its own when that copy has fallen behind or been edited, recording the CLI version it came from in an adjacent `.observal-extension.json`. A file Observal did not write is reported and left alone. If `npm:observal-pi` is registered in `~/.pi/agent/settings.json`, that takes precedence: nothing is installed locally, and a local copy Observal recognises as its own is removed (kept as `observal.ts.bak`) so Pi does not load the extension twice. A local file Observal did not write is left alone. Restart Pi or run `/reload` afterwards. `doctor patch` refuses to run until `observal auth login` has written a server URL, although it does not contact the server. ### 5. Activate the agent inside Pi Inside a Pi session, run `/agent` and pick the pulled agent. See [Agent profiles and swapping](#agent-profiles-and-swapping). ### 6. Check what is installed ```bash observal scan --harness pi observal doctor ``` *** ## Config paths | Purpose | Project scope | User scope | | -------------------- | ---------------------------------------------------- | --------------------------------------------------- | | Agent rules | `AGENTS.md` (project root) | `~/.pi/agent/agents/{agent}/AGENTS.md` | | MCP config | `.pi/agents/{agent}/mcp.json` | `~/.pi/agent/agents/{agent}/mcp.json` | | Skill definition | `.pi/agents/{agent}/skills/{name}/SKILL.md` | `~/.pi/agent/agents/{agent}/skills/{name}/SKILL.md` | | Active agent rules | `AGENTS.md` | `~/.pi/agent/AGENTS.md` | | Active MCP config | `.pi/mcp.json` | `~/.pi/agent/mcp.json` | | Active skills | `.pi/skills/{name}/SKILL.md` | `~/.pi/agent/skills/{name}/SKILL.md` | | Guidance files | `AGENTS.md`, `.pi/SYSTEM.md`, `.pi/APPEND_SYSTEM.md` | `~/.pi/agent/AGENTS.md` | | Telemetry extension | – | `~/.pi/agent/extensions/observal.ts` | | Pi settings | – | `~/.pi/agent/settings.json` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Observal lockfile | `~/.observal/lockfile.json` | `~/.observal/lockfile.json` | | Acknowledged cursors | `~/.observal/sync_state.json` | `~/.observal/sync_state.json` | | Pending batches | `~/.observal/pi_session_outbox/` | `~/.observal/pi_session_outbox/` | Pi MCP configs use the `mcpServers` key. *** ## Agent profiles and swapping Because Pi reads a single `AGENTS.md`, `mcp.json`, and `skills/` directory, only one Observal agent can be active at a time. In user scope, `observal agent pull` does not touch the active files. It writes into the per-agent profile directory, and the extension's `/agent` command makes a profile active: 1. On first use, `/agent` backs up the current `AGENTS.md`, `SYSTEM.md`, `mcp.json`, `skills/`, and `sandboxes/` into `~/.pi/agent/agents/default/`. 2. It removes the active `AGENTS.md`, `SYSTEM.md`, `mcp.json`, `skills/`, and `sandboxes/`, then copies in whichever of those the chosen profile contains. Items the profile lacks stay absent until you swap back to `default`. 3. When Observal credentials are configured, it records the chosen agent as `active_agent` in `~/.observal/config.json` and refreshes the layer snapshot. 4. It offers to reload the session so the new system prompt takes effect. Run `/agent` with no argument to pick from installed profiles, or `/agent ` to swap directly. Choose `default` to restore the backed-up configuration and clear the active agent. `/agent` lists only user-scope profiles under `~/.pi/agent/agents/`. *** ## Extension spec Observal installs a TypeScript extension named `observal.ts`. It has no runtime dependencies beyond `node:*` built-ins and is fail-open: telemetry failures never interrupt Pi. Ingest and checkpoint calls time out after five seconds, and the layer-snapshot upload after ten. | Pi event | Observal use | | ------------------ | ---------------------------------------------------------------------------------------------------------------------- | | `session_start` | Load config and cursors, upload the layer snapshot, recover stale sessions on startup, show `● observal` in the footer | | `agent_end` | Push new session lines after each turn | | `session_shutdown` | Push remaining lines and finalize the session | The extension also registers two commands, `/agent` and `/obs-sync`: | Command | Description | | ------------------ | ---------------------------------------- | | `/agent [name]` | Swap the active Observal agent profile | | `/obs-sync` | Show lines pushed and the server URL | | `/obs-sync flush` | Push pending lines now | | `/obs-sync config` | Show the config file path and server URL | `doctor patch` compares the installed file with the bundled source and replaces stale copies. *** ## Attribution Pi does not expose an Observal agent id in its session file. The extension resolves attribution from Observal's own state: 1. `observal agent pull` records the agent name, id, version, scope, pull time, and directory under the `pi` harness in `~/.observal/lockfile.json`. 2. The pull also records that agent as `active_agent` in `~/.observal/config.json`, and `/agent` updates the entry whenever you swap profiles. 3. On `session_start`, the extension reads `active_agent` and looks it up in the lockfile entry for the configured server URL, matching by id first and then by name. 4. The session payload is sent with the resolved `agent_id` and `agent_version`. 5. If there is no `active_agent`, or it has no `id`, the session is sent with a null agent id. If there is an `active_agent` but no lockfile match, the stored id is sent as-is; no directory-based guessing takes place. *** ## Session push behavior The Pi extension implements the same acknowledged delivery contract as the Python harnesses: 1. Resolve the session JSONL file and id from Pi's session manager. 2. Read complete records after the acknowledged byte and line cursor in `~/.observal/sync_state.json`. 3. Persist each pending batch under `~/.observal/pi_session_outbox/` before network delivery, in chunks of at most 500 lines. 4. Retry the batch idempotently until the server returns a contiguous acknowledgement covering it. 5. Advance the local cursor only to that acknowledged checkpoint. 6. On `session_shutdown`, send a SHA-256 audit manifest and replay any range the server asks to repair. 7. On the next startup, retry every pending outbox batch, then re-push unfinished sessions from the current project that changed within the last seven days until five have been finalized. Attempts that fail do not count toward the five. A missing or corrupt cursor is rebuilt from the server checkpoint on the next push. Pi's pending batches are the files under `~/.observal/pi_session_outbox/`. `observal ops telemetry status` reports server ingest health and the Python exporters' outbox; it does not count this directory. *** ## Agent profile format Observal writes the agent's rules as plain Markdown into `AGENTS.md`. Pi loads that file as the system prompt, so the whole file is the agent: ```markdown # Reviewer You are a code reviewer with the following specialization... ``` There is no frontmatter and no model field. `observal agent pull` does not write a model into Pi's configuration; the model is chosen inside Pi. *** ## Skill file format Pi skills live at: | Scope | Path | | ------- | ------------------------------------ | | Project | `.pi/skills/{name}/SKILL.md` | | User | `~/.pi/agent/skills/{name}/SKILL.md` | Example: ```markdown --- description: "Runs the project test suite" --- # Run Tests Run `pytest -q` from the project root. ``` When Codex is also installed, the bundled Observal skills are written once to the shared `~/.agents/skills/{name}/SKILL.md` location, which both harnesses read, instead of being duplicated under `~/.pi/agent/skills/`. *** ## Caveats **One active agent at a time.** A user-scope pull does not activate the agent. Run `/agent` inside Pi, or copy the profile into `~/.pi/agent/` by hand. `/agent` does not see project-scope profiles under `.pi/agents/`; copy their `mcp.json` and `skills/` into `.pi/` as shown in Setup step 3. **`/agent` replaces the active files.** The first swap backs up your existing `AGENTS.md`, `SYSTEM.md`, `mcp.json`, `skills/`, and `sandboxes/` into the `default` profile. Later swaps remove those five items and install only what the chosen profile contains, so a profile without `SYSTEM.md` leaves Pi with no `SYSTEM.md`. Edit the profile directory, not the active files, if you want changes to survive a swap. **The extension is shared per Pi install.** It is installed by `doctor patch`, not by each agent pull, and lives only in user scope. **Guidance files are scanned, with two exceptions.** Observal layers `AGENTS.md`, `.pi/SYSTEM.md`, and `.pi/APPEND_SYSTEM.md` as context. Scanning and `observal agent pull` never rewrite the project's `.pi/SYSTEM.md` or `.pi/APPEND_SYSTEM.md`. A project-scope pull does write the project's `AGENTS.md`, because that file is the agent's rules in Pi, and `/agent` swaps do replace the user-scope `~/.pi/agent/SYSTEM.md` as described above. **Attribution depends on the lockfile and `/agent`.** A profile copied by hand does not update `active_agent`, so its sessions keep whatever binding `~/.observal/config.json` already holds, or carry no agent id if it has none. **MCP config is Pi-specific.** Pi uses `mcp.json` with the `mcpServers` key under `.pi/` or `~/.pi/agent/`, not Claude Code or Kiro MCP paths. # Environment variables Complete reference for security-relevant server and CLI environment variables. Defaults are in `.env.example` and built into the CLI. ## File-backed values Every credential listed with file support accepts `NAME_FILE=/path/to/secret` instead of `NAME=value`. Files must be UTF-8, are limited to 64 KiB, and may end with one newline. Configuring both forms is an error. The server-package installer writes generated credentials to operator-owned files under `secrets/`, with read access limited to the configured container service group. ## Server (`observal-server`) ### Core / security | Variable | Default | Description | | ------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------- | | `SECRET_KEY` | `change-me-to-a-random-string` | Application encryption secret. Supports `SECRET_KEY_FILE`. Generate a random value of at least 32 characters. | | `OLD_SECRET_KEY` | unset | Previous secret during encrypted-setting rotation. Supports `OLD_SECRET_KEY_FILE`. | | `FRONTEND_URL` | `http://localhost:3000` | External frontend URL (OAuth redirects, email links) | | `CORS_ALLOWED_ORIGINS` | `http://localhost:3000` | Comma-separated allowed CORS origins | | `MAX_REQUEST_SIZE_MB` | `10` | Maximum request body size for ordinary endpoints | | `MAX_MIGRATION_REQUEST_SIZE_MB` | `6144` | Maximum combined multipart request size for Admin migration import and validation uploads | | `RATE_LIMIT_AUTH` | `10/minute` | General auth-endpoint rate limit | | `RATE_LIMIT_AUTH_STRICT` | `5/minute` | Login and password-reset rate limit | ### Databases | Variable | Default | Description | | ------------------------ | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `DATABASE_URL` | `postgresql+asyncpg://postgres:postgres@localhost:5432/observal` | PostgreSQL endpoint and source URL for `server migrate`. Supports `DATABASE_URL_FILE` in server configuration. | | `POSTGRES_USER` | `postgres` | Postgres container user | | `POSTGRES_PASSWORD` | container-specific | Direct Postgres password for existing installs | | `POSTGRES_PASSWORD_FILE` | generated by server-package setup | PostgreSQL native password-file input | | `CLICKHOUSE_URL` | `clickhouse://localhost:8123/observal` | ClickHouse endpoint and source URL for `server migrate`. Supports `CLICKHOUSE_URL_FILE` in server configuration. | | `TARGET_DATABASE_URL` | unset | Target PostgreSQL URL for `server migrate` import and validation commands | | `TARGET_CLICKHOUSE_URL` | unset | Target ClickHouse URL for `server migrate` import and validation commands | | `CLICKHOUSE_USER` | `default` | ClickHouse user | | `CLICKHOUSE_PASSWORD` | container-specific | Direct ClickHouse password for existing installs | | `REDIS_URL` | `redis://localhost:6379` | Redis connection string. Supports `REDIS_URL_FILE`. | | `DATA_RETENTION_DAYS` | `90` | ClickHouse TTL in days. `0` disables. Minimum non-zero: `7` | ### SSO OIDC, SAML, and SSO-only mode are configured in **Admin → SSO** and stored in dynamic settings. OIDC client changes require an API restart. Existing direct `OAUTH_*`, `SSO_ONLY`, and `SAML_*` values are imported once when the matching database setting is empty. File-backed forms such as `OAUTH_CLIENT_SECRET_FILE`, `GOOGLE_OAUTH_CLIENT_SECRET_FILE`, `GITHUB_OAUTH_CLIENT_SECRET_FILE`, `INSIGHTS_API_KEY_FILE`, `SAML_IDP_X509_CERT_FILE`, and `SAML_SP_KEY_ENCRYPTION_PASSWORD_FILE` remain in memory and are not imported into PostgreSQL or Redis. `SAML_SP_PRIVATE_KEY_FILE` and `SAML_SP_X509_CERT_FILE` must be configured together and override generated database SP material. ### JWT signing | Variable | Default | Description | | ----------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------- | | `JWT_SIGNING_ALGORITHM` | `ES256` | `ES256` or `RS256` | | `JWT_KEY_DIR` | `~/.observal/keys` (outside Docker) / `/data/keys` (Docker) | Directory for generated signing keys. **Back this up.** | | `JWT_KEY_PASSWORD` | unset | Optional private-key encryption password. Supports `JWT_KEY_PASSWORD_FILE`. | ### AWS (Bedrock) > **Note:** Bedrock now supports [API keys](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) — generate one from the AWS console and use it like any other provider. See [Insights LLM Setup](https://github.com/Observal/Observal/blob/main/docs/insights-setup.md). These environment variables are **not required** if you use Bedrock API keys (recommended). They exist only for legacy setups using instance roles or ECS task roles where LiteLLM auto-discovers credentials from the environment. | Variable | Default | Description | | ----------------------- | ----------- | ------------------------------------------- | | `AWS_ACCESS_KEY_ID` | - | Only for legacy IAM auth (not recommended) | | `AWS_SECRET_ACCESS_KEY` | - | Only for legacy IAM auth (not recommended) | | `AWS_SESSION_TOKEN` | - | Temporary credentials (STS AssumeRole) | | `AWS_REGION_NAME` | `us-east-1` | AWS region (used by LiteLLM's boto3 client) | ### Git operations (submission analysis) | Variable | Default | Description | | ---------------------- | ---------------- | ------------------------------------------------------------------------------- | | `ALLOW_INTERNAL_URLS` | `false` | Allow internal/private Git URLs (for GitLab / GHE) | | `GIT_CLONE_TOKEN` | - | Auth token for private repos. Supports `GIT_CLONE_TOKEN_FILE`. | | `GIT_CLONE_TOKEN_USER` | `x-access-token` | Token username: `x-access-token` (GitHub), `oauth2` or `private-token` (GitLab) | | `GIT_CLONE_TIMEOUT` | `120` | Clone timeout, seconds | ### Demo accounts (seeded on first startup if no users exist) | Variable | Default | | --------------------------- | ----------------------------------------------------------------------- | | `DEMO_SUPER_ADMIN_EMAIL` | `super@demo.example` | | `DEMO_SUPER_ADMIN_PASSWORD` | `super-changeme` in source Compose; generated file in server package | | `DEMO_ADMIN_EMAIL` | `admin@demo.example` | | `DEMO_ADMIN_PASSWORD` | `admin-changeme` in source Compose; generated file in server package | | `DEMO_REVIEWER_EMAIL` | `reviewer@demo.example` | | `DEMO_REVIEWER_PASSWORD` | `reviewer-changeme` in source Compose; generated file in server package | | `DEMO_USER_EMAIL` | `user@demo.example` | | `DEMO_USER_PASSWORD` | `user-changeme` in source Compose; generated file in server package | Each demo password supports its corresponding `_FILE` form. Remove demo variables and files before a real deployment. ### Docker host ports Used only by Docker Compose. Prometheus and Grafana ports apply only when `docker-compose.observability.yml` is included. Remap if a default is already in use. | Variable | Default | Service | | ----------------------- | --------------------------------- | ------------------------------------------------------------------------------------ | | `OBSERVAL_BIND_ADDRESS` | `127.0.0.1` in the server package | Host address for published ports. A non-loopback value is explicit remote exposure. | | `OBSERVAL_SECRET_GID` | operator group from `setup.sh` | Supplementary container group allowed to read server-package files with mode `0640`. | | `API_HOST_PORT` | `8000` | API (internal, behind LB) | | `WEB_HOST_PORT` | `3000` | Web UI | | `POSTGRES_HOST_PORT` | `5432` | Postgres | | `CLICKHOUSE_HOST_PORT` | `8123` | ClickHouse | | `REDIS_HOST_PORT` | `6379` | Redis | | `PROMETHEUS_HOST_PORT` | `9090` | Prometheus, optional | | `GRAFANA_HOST_PORT` | `3001` | Grafana, optional | ### Grafana Only used when the optional Grafana overlay or Terraform `observability_stack = "grafana"` is enabled. | Variable | Default | Description | | ---------------------------------- | -------------------------------- | ------------------------------------------------------------- | | `GRAFANA_ADMIN_USER` | `admin` | Grafana admin username | | `GRAFANA_ADMIN_PASSWORD` | `admin` | Direct Grafana admin password for existing installs | | `GRAFANA_ADMIN_PASSWORD_FILE` | generated file in server package | Grafana administrator password file | | `GRAFANA_CLICKHOUSE_PASSWORD_FILE` | generated file in server package | ClickHouse datasource password file mounted only into Grafana | ## CLI (`observal-cli`) Read from the environment at invocation time. Override values in `~/.observal/config.json` per invocation. | Variable | Default | Description | | --------------------------- | ------------------------------ | -------------------------------------------------------------- | | `OBSERVAL_SERVER_URL` | from `~/.observal/config.json` | Server URL | | `OBSERVAL_ACCESS_TOKEN` | from `~/.observal/config.json` | Access token (preferred for CI) | | `OBSERVAL_API_KEY` | from `~/.observal/config.json` | API key alias for `OBSERVAL_ACCESS_TOKEN` | | `OBSERVAL_TOKEN` | from `~/.observal/config.json` | CI token alias with highest legacy precedence | | `OBSERVAL_PASSWORD` | unset | Password for credential or bootstrap login | | `OBSERVAL_CURRENT_PASSWORD` | unset | Current password for non-interactive password changes | | `OBSERVAL_NEW_PASSWORD` | unset | New password for non-interactive or mandatory password changes | | `OBSERVAL_TIMEOUT` | `30` | HTTP timeout in seconds | The token and password variables support corresponding `_FILE` forms. Configuring both a direct variable and its file form is an error. When multiple token aliases are present, the legacy precedence remains `OBSERVAL_TOKEN`, then `OBSERVAL_API_KEY`, then `OBSERVAL_ACCESS_TOKEN`. Example CI usage: ```bash export OBSERVAL_SERVER_URL=https://observal.your-company.internal export OBSERVAL_API_KEY= observal ops traces --limit 100 --output json | jq ``` ## Related * [Self-Hosting → Configuration](/self-hosting/configuration), narrative view, grouped by concern * [Config files](/reference/config-files), `~/.observal/` file layout # API endpoints REST and GraphQL surface of the Observal server. Unless noted, endpoints require authentication via Bearer token or API key (`Authorization: Bearer ` or `X-API-Key: `). Base path: `/api/v1`. ## Auth | Method | Path | Description | | ------ | ---------------------- | ---------------------------------------------------------------------- | | `POST` | `/auth/bootstrap` | Auto-create admin on fresh server (localhost only) | | `POST` | `/auth/register` | Self-registration (email + password; `deployment.sso_only=false` only) | | `POST` | `/auth/login` | Login with API key or email + password | | `POST` | `/auth/exchange` | Exchange one-time OAuth code for credentials | | `GET` | `/auth/whoami` | Current user info | | `POST` | `/auth/token` | Exchange credentials for JWT access + refresh tokens | | `POST` | `/auth/token/refresh` | Rotate refresh token for new access token | | `POST` | `/auth/token/revoke` | Revoke a refresh token | | `POST` | `/auth/request-reset` | Request password reset (code logged to server console) | | `POST` | `/auth/reset-password` | Reset password with code + new password | | `GET` | `/auth/oauth/login` | Initiate OAuth SSO flow | | `GET` | `/auth/oauth/callback` | OAuth callback handler | ## Registry Per type: `mcps`, `agents`, `skills`, `hooks`, `prompts`, `sandboxes`. All `{id}` parameters accept a UUID or a name. | Method | Path | Description | | -------- | ---------------------- | ------------------------------------ | | `POST` | `/{type}` | Submit / create | | `GET` | `/{type}` | List approved items | | `GET` | `/{type}/{id}` | Get details | | `POST` | `/{type}/{id}/install` | Get harness config snippet | | `DELETE` | `/{type}/{id}` | Delete | | `GET` | `/{type}/{id}/metrics` | Metrics | | `POST` | `/agents/{id}/pull` | Pull agent (installs all components) | `POST /{type}/{id}/install` accepts `version` for agents, MCP servers, skills, and hooks. The response reports the `version` that was installed; component installs also return its `version_id` and content `digest`. ### Agent versions and locks Each agent version pins exact component versions. Installs generate every component from its pinned version. Agent versions released before pinning can have unlocked components: those install at their latest approved release, are reported with `source: fallback-latest` in the install `lock`, and are refused by a `strict` install. | Method | Path | Description | | ------ | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/agents/{id}/versions` | Version history | | `POST` | `/agents/{id}/versions` | Release a version for review. Components keep the current release's pins unless a component names a `version` or `refresh_components` is true | | `GET` | `/agents/{id}/versions/{version}/lock` | The version's lock: exact component versions, version ids, and digests. Frozen once approved | | `GET` | `/agents/{id}/versions/{version}/outdated` | For each pin, the latest approved release and whether the pin is behind | | `PUT` | `/agents/{id}` | Edits version-owned fields only while the latest version is a draft, pending, or rejected; otherwise `409` | A component reference's `version` must name an approved (or archived) release. Drafts may also pin a release that is not approved yet, but only a caller who may read it (its owner, co-authors, admins, and reviewers); to anyone else it does not exist, as on the component version routes. `POST /agents/{id}/install` also accepts `strict`. Its response includes `version` (the agent version installed) and `lock`: `status` (`locked`, `partial`, or `unlocked`), `digest`, `components` with each one's `source`, and `problems`. With `strict: true`, any problem is a `409` and nothing is generated. ### Scan | Method | Path | Description | | ------ | ------- | -------------------------------------------- | | `POST` | `/scan` | Bulk register items from harness config scan | ### Review | Method | Path | Description | | ------ | ---------------------- | ------------------------ | | `GET` | `/review` | List pending submissions | | `GET` | `/review/{id}` | Submission details | | `POST` | `/review/{id}/approve` | Approve | | `POST` | `/review/{id}/reject` | Reject | ## Discovery (ARD) Agentic Resource Discovery endpoints. Authentication is optional: anonymous callers see public, approved resources only when `deployment.public_registry_enabled` is on; authenticated callers see what the registry's visibility rules already grant them (public, their teams', and their own drafts). Errors use the ARD envelope `{"errorCode": "INVALID_ARGUMENT", "message": "..."}`. See [ADR 0001](/architecture-decisions/0001-agentic-resource-discovery). | Method | Path | Description | | ------ | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/.well-known/ard.json` | Manifest of the registry entry plus this deployment's public approved resources (not imported remote agents) | | `GET` | `/.well-known/ai-catalog.json` | Predecessor path, same document | | `POST` | `/api/v1/ard/search` | ARD Search. `score` is relevance only; `obs:approval`, `obs:availability`, `obs:supportedHarnesses`, `obs:delegable` carry the rest; remote A2A results add `obs:provider` from the card | | `GET` | `/api/v1/ard/agents` | ARD List: deterministic browse with `filter`, `orderBy`, `pageSize`, `pageToken` | | `POST` | `/api/v1/ard/explore` | ARD Explore: `501` until facets ship | | `GET` | `/api/v1/ard/entries/{identifier}` | One complete entry by `urn:air:` identifier (Observal-specific) | | `GET` | `/api/v1/artifacts/{kind}/{uuid}/{version}` | Permanent versioned artifact; `Digest` and `X-Artifact-Digest` headers | Search request body follows the spec: `{"query": {"text": "...", "filter": {...}}, "federation": "none", "pageSize": 5}`. Supported filter terms: `type`, `tags`, `capabilities`, `publisher`, `version`, `obs:kind`, `obs:supportedHarnesses`, `obs:lifecycle`, `obs:activatable`. ### Remote A2A agents Remote agents are registered by Agent Card URL and reviewed before anyone can delegate to them. They then appear in search as `type: application/a2a-agent-card+json` with the reviewed card pinned in the entry (`obs:agentCard`, `obs:a2aInterface`). Clients call the agent directly; the server does not proxy A2A traffic. See [ADR 0002](https://github.com/BlazeUp-AI/Observal/tree/main/docs/adr/0002-a2a-delegation.md). | Method | Path | Description | | -------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET` | `/api/v1/ard/imports` | Remote agents the caller can see, including their own and (reviewers) pending ones | | `POST` | `/api/v1/ard/imports/a2a` | Register or refresh a card: `{"cardUrl": "...", "visibility": "private\|team\|public", "teamId": "..."}`. `201` new, `200` refresh; a changed card returns to `pending` | | `POST` | `/api/v1/ard/imports/{identifier}/review` | Reviewers in scope (team owners and reviewers for team agents, global reviewers for public, admins for private): `{"action": "approve\|reject", "reason": "...", "digest": "sha256:..."}` (`digest` of the reviewed card required to approve, 409 when the card changed since; reason required to reject) | | `DELETE` | `/api/v1/ard/imports/{identifier}` | Owner or admin: remove from discovery | Cards must be served over https from a public address. Internal hosts are allowed individually with the `discovery.a2a_private_hosts` setting. The `discovery.delegation_enabled` setting (default on) adds the `observal-agents` MCP server to every pulled Agent. ## Telemetry | Method | Path | Description | | ------ | ---------------------------- | ---------------------------------------------------------------------------------------------------- | | `POST` | `/telemetry/events` | Legacy event ingestion | | `GET` | `/telemetry/status` | Data flow status | | `POST` | `/ingest/session` | Idempotently ingest indexed session source records and return the highest contiguous acknowledgement | | `GET` | `/ingest/session/checkpoint` | Get the caller's contiguous line/byte checkpoint for a harness session | Session record identity is scoped by project, user, harness, session ID, and source line index. Retrying the same content at the same index is safe; different content at an already acknowledged index returns `409`. Final session requests can include `session_hash` and `hashed_line_count`. The response includes `integrity_ok`, `server_hash`, and `repair_from_line`. A failed audit rewinds the durable checkpoint to the first affected range; the exporter then replays that range idempotently. Hashing and canonical manifest scans occur only for final/audit requests, not incremental ingest. ## Telemetry hooks | Method | Path | Description | | ------ | ------------------ | ------------------------------------------------------- | | `POST` | `/telemetry/hooks` | Ingest lifecycle hook events (used by Kiro shell hooks) | ## Alerts | Method | Path | Description | | -------- | ---------------------- | -------------------- | | `GET` | `/alerts` | List alert rules | | `POST` | `/alerts` | Create alert rule | | `PATCH` | `/alerts/{id}` | Update alert rule | | `DELETE` | `/alerts/{id}` | Delete alert rule | | `GET` | `/alerts/{id}/history` | Alert firing history | ## Feedback | Method | Path | Description | | ------ | ------------------------ | -------------- | | `POST` | `/feedback` | Submit rating | | `GET` | `/feedback/{type}/{id}` | Get feedback | | `GET` | `/feedback/summary/{id}` | Rating summary | ## Admin Requires `admin` or `super_admin` role. | Method | Path | Description | | ------ | ------------------------------------ | --------------------------- | | `GET` | `/admin/settings` | List settings | | `PUT` | `/admin/settings/{key}` | Set a value | | `GET` | `/admin/users` | List users | | `POST` | `/admin/users` | Create user | | `PUT` | `/admin/users/{id}/role` | Change role | | `PUT` | `/admin/users/{id}/password` | Reset user password (admin) | | `GET` | `/admin/weights` | Get dimension weights | | `PUT` | `/admin/weights` | Set dimension weights | | `GET` | `/admin/canaries/{agent_id}/reports` | Canary detection reports | ## GraphQL Single endpoint, query + subscription via WebSocket. | Path | Description | | ----------------- | ---------------------------- | | `/api/v1/graphql` | Session update subscriptions | Subscriptions use `graphql-ws` protocol. ## Health | Method | Path | Description | | ------ | ---------- | ----------------------------------- | | `GET` | `/health` | Readiness - checks DB + ClickHouse | | `GET` | `/healthz` | Liveness - is the API process alive | ## Rate limiting Auth endpoints are subject to `RATE_LIMIT_AUTH` and `RATE_LIMIT_AUTH_STRICT`. Non-auth endpoints are not rate-limited by default; put a reverse proxy or API gateway in front if you need it. ## Request size limits `MAX_REQUEST_SIZE_MB` (default `10`) caps body size on all endpoints. Large telemetry batches may need tuning. ## Related * [Authentication and SSO](/self-hosting/authentication) * [Hooks specification](/reference/hooks-spec) # Hooks specification The schema Observal uses for hook definitions -- both the registry hook type (`observal registry hook`) and hooks wired into harness configs by `observal agent pull` / `observal doctor patch`. Current version: `HOOKS_SPEC_VERSION = "5"` (see `observal_cli/hooks_spec.py`). ## Where hooks live Two distinct things share the name "hook": 1. **Registry hooks**: packaged, versioned hook definitions in the Observal registry. Install them via `observal registry hook install`. 2. **harness hooks**: entries in `~/.claude/settings.json`, `.kiro/agents/.json`, etc. These are written by `observal agent pull` and `observal doctor patch`. Both use the same event vocabulary. ## Events | Event | When it fires | | ------------------ | ----------------------------------------- | | `SessionStart` | New harness session begins | | `Stop` | Session ends | | `SubagentStop` | Sub-agent session ends (Claude Code only) | | `UserPromptSubmit` | User submits a prompt | | `PreToolUse` | Before a tool call | | `PostToolUse` | After a tool call (with result) | | `Notification` | harness surfaces a notification | Source: `observal_cli/constants.py:VALID_HOOK_EVENTS`. ## Handler types | Type | Payload | Used by | | --------- | --------------------------------- | ---------------------------------------------------- | | `command` | Shell command with templated args | Kiro (shell hooks only), Claude Code (local scripts) | | `http` | URL + method + headers + body | Claude Code (native HTTP hooks) | ## Execution modes | Mode | Semantics | | ---------- | ----------------------------------------------------- | | `async` | Fire and forget - harness doesn't wait | | `sync` | harness waits for handler to return before continuing | | `blocking` | Handler can veto the event (e.g. block a tool call) | Source: `observal_cli/constants.py:VALID_HOOK_EXECUTION_MODES`. ## Scopes | Scope | Effect | | --------- | ------------------------------------- | | `agent` | Applies only to one agent | | `session` | Applies for the duration of a session | | `global` | Applies across everything | ## Metadata marker Observal writes a `_observal` key into hook matcher groups so subsequent runs of `doctor patch` / `pull` can find and update only Observal-managed hooks without stomping on user-authored ones. ```json { "_observal": { "version": "5", "source": "observal-scan", "installed_at": "2026-04-21T14:30:55Z" }, "PreToolUse": [ ... ] } ``` Older installs (pre-metadata) are detected with a fallback heuristic. ## Claude Code: native HTTP hook example ```json { "hooks": { "PreToolUse": [ { "_observal": { "version": "5", "source": "observal-pull" }, "matcher": "*", "type": "http", "url": "http://localhost/api/v1/telemetry/hooks", "method": "POST", "headers": { "Authorization": "Bearer ${OBSERVAL_API_KEY}", "Content-Type": "application/json" } } ] } } ``` ## Kiro: shell-command hook example Kiro doesn't support native HTTP hooks, so Observal uses `curl`: ```json { "name": "my-agent", "hooks": { "agentSpawn": "curl -s -X POST http://localhost/api/v1/telemetry/hooks -H 'Authorization: Bearer $OBSERVAL_API_KEY' -H 'Content-Type: application/json' -d @-", "userPromptSubmit": "curl -s -X POST http://localhost/api/v1/telemetry/hooks ...", "preToolUse": "curl -s -X POST http://localhost/api/v1/telemetry/hooks ...", "postToolUse": "curl -s -X POST http://localhost/api/v1/telemetry/hooks ...", "stop": "curl -s -X POST http://localhost/api/v1/telemetry/hooks ..." } } ``` ## Event name mapping (Claude Code ↔ Kiro) Kiro uses camelCase / lowercase event names; Claude Code uses PascalCase. Observal maps between them. | Claude Code | Kiro | | ------------------ | ------------------ | | `SessionStart` | `agentSpawn` | | `Stop` | `stop` | | `SubagentStop` | *(no equivalent)* | | `UserPromptSubmit` | `userPromptSubmit` | | `PreToolUse` | `preToolUse` | | `PostToolUse` | `postToolUse` | | `Notification` | *(no equivalent)* | ## Registry hook payload shape When submitting a hook to the registry (`observal registry hook submit`): ```json { "name": "pretooluse-logger", "description": "Logs every tool call to a local file", "event": "PreToolUse", "handler_type": "command", "command": "echo \"$TOOL_NAME $(date)\" >> ~/.observal/tool-log.txt", "execution_mode": "async", "scope": "agent", "harness": ["claude-code", "kiro"] } ``` Each field is validated server-side against the lists in `observal_cli/constants.py` (mirrored from `observal-server/schemas/constants.py`). ## Source of truth * `observal_cli/hooks_spec.py`: version, metadata marker, spec shape * `observal_cli/constants.py`: valid events, handler types, execution modes, scopes * `observal-server/schemas/constants.py`: server-side mirror A sync test (`tests/test_constants_sync.py`) ensures CLI and server stay in lockstep. ## Related * [`observal registry hook`](/cli-reference/registry) * [Session tracking and reconciliation](/core-concepts/session-tracking) # Config files Every file Observal reads or writes on the client (`~/.observal/`) and in each harness's config directory. ## Client-side: `~/.observal/` | File | Purpose | Permissions | | -------------------------- | --------------------------------------------------------------------------------------------- | ---------------- | | `config.json` | CLI config (server URL, access token, user info, timeout) | `0600` | | `aliases.json` | User-defined shortcuts (`@my-mcp` → UUID) | `0600` | | `last_results.json` | Latest typed list output, enabling row-number references | `0600` | | `telemetry_buffer.db` | Durable SQLite outbox for Python session exporters awaiting contiguous server acknowledgement | `0600` | | `opencode_session_outbox/` | Per-session durable OpenCode plugin batches and acknowledged line state | `0600` files | | `pi_session_outbox/` | Durable pending Pi extension batches | `0600` files | | `sync_state.json` | Acknowledged byte/line cursors for file-backed exporters | owner read/write | | `keys/` | Server-side JWT keys (operators only; path controlled by `JWT_KEY_DIR`) | `0600` | ### `config.json` schema ```json { "server_url": "https://observal.your-company.internal", "access_token": "ey...", "refresh_token": "ey...", "user_id": "f9f3...", "user_name": "Alice", "username": "alice", "timeout": 30, "update_check": true, "update_check_interval": 86400, "update_check_repo": "" } ``` Authentication and identity fields are managed by `observal auth`. The `config set` command accepts only `server_url`, `timeout`, `update_check`, `update_check_interval`, and `update_check_repo`. Supported environment variables override persisted values for the current invocation; see [Environment variables](/reference/environment-variables). ### Durable session outbox Python session exporters persist each observed batch in `telemetry_buffer.db` before network delivery. OpenCode and Pi use per-session files under their native outbox directories because their TypeScript runtimes cannot call the Python SQLite engine. All follow the same protocol: pending data survives process restarts and failed attempts, and the source line advances only when the server's contiguous checkpoint covers the complete batch. Observal does not silently evict unacknowledged records at capacity. `sync_state.json` is a cache of acknowledged local positions, not the authority for delivered history. If it is missing, corrupt, or stale, recovery validates and restores positions from the authenticated server checkpoint. Finalized sessions also send a SHA-256 audit manifest; hashing is not performed on ordinary incremental uploads. Use `observal ops telemetry status` to inspect pending batch count, disk use, oldest pending time, and last successful acknowledgement. ### `aliases.json` schema ```json { "my-mcp": "498c17ac-1234-4567-89ab-cdef01234567", "reviewer": "alice/reviewer" } ``` Use anywhere that accepts a compatible reference by prefixing the alias with `@`. ### `last_results.json` schema ```json { "item_type": "skill", "ids": ["498c17ac-1234-4567-89ab-cdef01234567"], "names": { "reviewer": "498c17ac-1234-4567-89ab-cdef01234567" } } ``` Each list invocation replaces this cache, including an empty result. Numeric row references are valid only for the component type that produced the latest list. ## harness-side ### Claude Code | Path | Purpose | | ------------------------------ | ------------------------------------ | | `~/.claude/settings.json` | Hooks, MCP servers, telemetry config | | `~/.claude/agents/.json` | User-scoped sub-agent definitions | | `.claude/agents/.json` | Project-scoped sub-agent definitions | | `.claude/skills//` | Installed skills (SKILL.md + assets) | | `AGENTS.md` / `CLAUDE.md` | Rules loaded into context | ### Kiro | Path | Purpose | | ---------------------------- | ------------------------------------------------------------------------------ | | `.kiro/settings/mcp.json` | Project-level MCP servers with direct commands or URLs | | `~/.kiro/settings/mcp.json` | Global MCP servers | | `.kiro/agents/.json` | Project-level agent config with telemetry hooks | | `~/.kiro/agents/.json` | Global agent config | | `.kiro/steering/.md` | Steering files (system instructions with YAML frontmatter for inclusion modes) | | `.kiro/skills/` | Kiro skills (SKILL.md) | | `.kiro/hooks/` | Standalone hook definitions | | `AGENTS.md` | Rules loaded into context (compat with Claude Code) | ### Cursor | Path | Purpose | | ------------------ | ---------------------------------------- | | `.cursor/mcp.json` | MCP servers with direct commands or URLs | | `.cursor/rules/` | Cursor rules | | `AGENTS.md` | Rules | ### VS Code | Path | Purpose | | ------------------ | ---------------------------------------- | | `.vscode/mcp.json` | MCP servers with direct commands or URLs | | `AGENTS.md` | Rules loaded into context | ### Codex CLI | Path | Purpose | | ----------- | ------------------------------ | | `AGENTS.md` | Rules (rules-only integration) | ### Pi | Path | Purpose | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `~/.pi/agent/extensions/observal.ts` | Bundled telemetry extension, installed locally when npm isn't configured | | `~/.pi/agent/extensions/.observal-extension.json` | Manifest recording the CLI version the local extension was installed from | | `~/.pi/agent/extensions/observal.ts.bak` | Copy of the previous extension, kept whenever Observal replaces or removes one | | `~/.pi/agent/settings.json` | Pi's own config; a `packages` entry of `npm:observal-pi[@version]` here selects the npm installation mode instead | Two mutually exclusive installation modes are supported: * **npm**: `npm:observal-pi` is configured in `~/.pi/agent/settings.json` (with or without a pinned version, and whether or not Pi has downloaded it yet). Observal never installs `extensions/observal.ts` in this mode — `observal doctor` only reports when a pinned version is older than the installed CLI, with a `pi update npm:observal-pi` reminder. Pi loads a local `extensions/observal.ts` in addition to the npm package, so if one is left over from local mode every session is sent twice; `observal doctor patch` removes a local copy it recognizes as its own (keeping it as `observal.ts.bak`) and leaves a file Observal did not write alone. * **local**: no npm package is configured. `observal auth login` and `observal doctor patch` install the extension bundled with the CLI directly to `extensions/observal.ts`, recording the installed CLI version in the adjacent manifest. A later CLI upgrade rewrites each of those files atomically — the two are written in sequence, not as a single atomic unit — and prints a reminder to restart Pi or run `/reload`; a newer local install than the current CLI is left alone. If the file still carries the current version but its contents have drifted from the bundled source — an edited working copy, say — doctor restores it and keeps the previous contents as `observal.ts.bak`. Observal keeps a copy of the file whenever it rewrites or deletes one, named `observal.ts.bak` (`.bak.1`, `.bak.2`, … if that name is taken). The copy does not end in `.ts`, so Pi does not load it; delete it once you are satisfied with the result. There are two cases where no copy is made, because nothing is lost: adopting a file whose contents already match the bundle, where only the manifest is written, and `observal doctor cleanup` removing a tracked install that still matches what Observal wrote. Every other refresh, restore, migration and duplicate removal keeps one — a manifest records which CLI version wrote an install, not that its bytes are still unedited. ### Migrating installs from before version tracking CLI versions before the manifest existed wrote `extensions/observal.ts` with nothing beside it. Those installs are recognized by the doc header every Observal extension carries, and `observal auth login` or `observal doctor patch` migrates them: the existing file is copied to `observal.ts.bak` (`.bak.1`, `.bak.2`, … if that name is taken), the current extension is written in its place, and the manifest is created. The backup does not end in `.ts`, so Pi does not load it; delete it once you're satisfied with the upgrade. If `extensions/observal.ts` exists without a matching manifest, doesn't match the bundled source, and carries no Observal header, Observal treats it as unmanaged and never overwrites or removes it — remove or move the file aside, then re-run `observal doctor patch --harness pi` to let Observal manage it. ## Safe writes `observal doctor patch` and `observal doctor cleanup` write each managed configuration file atomically and preserve unrelated entries. Use `--dry-run --output json` to review planned harness changes before writing. Agent Pull reports every created or merged file in its result. ## File permissions Client-side files under `~/.observal/` are created with mode `0600` (owner read/write only). This holds your access token, so don't loosen the permissions. ## Related * [Environment variables](/reference/environment-variables) - supported runtime overrides * [`observal config`](/cli-reference/config), CLI surface for editing # Security assurance case ## Claim Observal protects registry control, authentication credentials, agent configuration, and telemetry against unauthorized disclosure or modification when operators follow the documented deployment requirements. This case records the threats, boundaries, controls, evidence, and residual risks supporting that claim. ## Assets The primary assets are: * User passwords, access tokens, refresh tokens, OAuth secrets, Git tokens, and signing keys * Registry agents and their MCP, skill, hook, prompt, and sandbox configurations * Session transcripts, traces, insights, audit events, and user identity data * Release artifacts, release tags, and the software supply chain * Administrative control over users, teams, policy, retention, and SSO ## Threat actors and assumptions Relevant actors include unauthenticated internet clients, malicious authenticated users, compromised user tokens, hostile registry submissions, compromised dependencies, and network attackers between clients and a remote deployment. Administrators and host operators are trusted to protect the host, secret files, backups, and TLS termination. PostgreSQL, ClickHouse, Redis, the API, and workers are trusted only inside the deployment network. A host-level compromise, malicious administrator, compromised identity provider, or compromised GitHub release workflow is outside the controls of the application alone. Recovery procedures and external infrastructure controls must address those events. ## Trust boundaries 1. **Client to frontend:** Untrusted browsers, CLIs, harness hooks, and integrations cross the HTTPS boundary into nginx and the API. Packaged HTTP listeners bind to loopback by default. Remote access requires an operator-selected TLS proxy or an explicit non-loopback bind. 2. **API authentication and authorization:** Requests cross from unauthenticated parsing into JWT, password, SSO, role, team, and ownership checks. 3. **Registry execution inputs:** URLs, archives, MCP metadata, prompts, and harness configuration cross from users or external repositories into scanners and config generation. 4. **Application to data services:** The API and workers cross the container network into PostgreSQL, ClickHouse, and Redis. Database ports bind to loopback in the server package by default. 5. **Secret storage:** Operator-controlled secret files and JWT or SAML key files cross into application memory. File-backed dynamic secrets are not copied into PostgreSQL or Redis. 6. **External services:** OAuth providers, source repositories, webhooks, and model providers cross outbound HTTPS and SSRF policy boundaries. 7. **Release supply chain:** Source and GitHub Actions identities cross into signed tags, built artifacts, provenance attestations, and public downloads. ## Security requirements and arguments ### Identity and least privilege Passwords use scrypt with a per-user salt. Tokens use configurable ES256 or RS256 asymmetric signing, short-lived access tokens, refresh-token revocation, and published JWKS. Role, team, visibility, ownership, and admin dependencies enforce authorization at API boundaries. Redis-backed revocation fails closed. Evidence: [`models/user.py`](https://github.com/BlazeUp-AI/Observal/tree/main/observal-server/models/user.py), [`services/crypto.py`](https://github.com/BlazeUp-AI/Observal/tree/main/observal-server/services/crypto.py), and [Authentication and SSO](/self-hosting/authentication). ### Credential and key protection Operator credentials support bounded `NAME_FILE` inputs, direct and file values cannot be configured together, and server-package secrets are operator-owned and readable only by the configured container service group. JWT keys are persisted separately with owner-only permissions. SAML SP keys and certificates can be mounted as dedicated files. File-backed admin values are reported as externally managed and cannot be overwritten or revoked through the admin API. End-user password hashes and issued-token metadata remain in their purpose-built database records. They are not deployer secrets and do not belong in shared host secret files. Evidence: [`observal_shared/secrets.py`](https://github.com/BlazeUp-AI/Observal/tree/main/packages/observal-shared/observal_shared/secrets.py), [Configuration](/self-hosting/configuration), and [Environment variables](/reference/environment-variables). ### Network confidentiality Remote deployments are expected to terminate TLS 1.2 or TLS 1.3 using Caddy, nginx, an enterprise load balancer, or a cloud ingress. Standard HTTPS clients retain certificate and hostname verification. The server package exposes HTTP and data-service ports only on loopback by default. A non-loopback plaintext listener requires an explicit bind setting. Evidence: [Single-node deployment](/self-hosting/single-node-deploy), [`docker/nginx.production.conf`](https://github.com/BlazeUp-AI/Observal/tree/main/docker/nginx.production.conf), and [Requirements](/self-hosting/requirements). ### Untrusted input containment Pydantic schemas, enums, length bounds, allowlists, namespace validation, and database constraints reject malformed input. Outbound URL paths use SSRF guards. Redirects, archive paths, XML, SVG, host headers, proxy headers, and request sizes receive dedicated validation. Containers add non-root execution, read-only filesystems, temporary filesystems, and `no-new-privileges` where supported. Evidence: [Trusted proxies and network security](https://github.com/BlazeUp-AI/Observal/tree/main/docs/self-hosting/trusted-proxies.md), [`services/ssrf_guard.py`](https://github.com/BlazeUp-AI/Observal/tree/main/observal-server/services/ssrf_guard.py), and [`docker/docker-compose.yml`](https://github.com/BlazeUp-AI/Observal/tree/main/docker/docker-compose.yml). ### Cryptographic and supply-chain integrity JWT signing can switch between ES256 and RS256 without recompilation. Algorithm changes retain old public verification keys for the token transition window and reject header or key-type confusion. Release artifacts carry GitHub keyless Sigstore provenance, and release tags are signed and verified with the release workflow's OIDC identity before publication. Evidence: [`services/crypto.py`](https://github.com/BlazeUp-AI/Observal/tree/main/observal-server/services/crypto.py), [Release verification](/security/release-verification), and [the release workflow](https://github.com/BlazeUp-AI/Observal/tree/main/.github/workflows/release.yml). ### Detection and recovery Security-sensitive administration emits audit and security events. Health checks, diagnostics, backup guidance, key backup guidance, retention controls, and vulnerability reporting support detection and recovery. Private vulnerability reports have documented acknowledgement and remediation targets. Evidence: [Security policy](https://github.com/BlazeUp-AI/Observal/tree/main/SECURITY.md), [Backup and restore](/self-hosting/backup-and-restore), and [Authentication and SSO](/self-hosting/authentication). ## Common implementation weakness mitigations | Weakness | Mitigation | | ------------------------------------ | ----------------------------------------------------------------------------------------------------- | | Injection and malformed data | Typed schemas, allowlists, parameterized SQLAlchemy queries, and output encoding | | Broken access control | Central role dependencies plus ownership, team, and visibility checks | | SSRF | Scheme and address validation before outbound requests | | Path traversal and unsafe archives | Resolved-path containment, archive member limits, and type checks | | Credential disclosure | Secret redaction, encrypted sensitive settings, restricted secret files, and support-bundle filtering | | Token forgery or algorithm confusion | Asymmetric signatures, key-derived allowlists, required key IDs, expiry, and revocation | | XML and browser attacks | SAML validation, XML protections, SVG sanitization, CSP, frame denial, and content-type controls | | Resource exhaustion | Request, line, archive, query, timeout, connection-pool, and rate limits | | Supply-chain substitution | Locked dependencies, dependency scanning, signed release tags, checksums, and artifact provenance | ## Residual risks * Explicit non-loopback plaintext mode exposes credentials and telemetry to network interception. It is intended only for trusted private networks or an upstream TLS terminator. * Loopback HTTP relies on the host boundary. Other privileged processes on a compromised host can observe local traffic and files. * File-backed secrets are only as secure as host permissions, mounts, backups, and the operator's rotation process. * A compromised administrator, host, identity provider, or canonical GitHub Actions workflow can exercise its trusted authority. * Third-party MCP servers, hooks, skills, models, and harnesses may process data outside Observal's boundary. Operators must review and isolate them appropriately. * Historical releases and tags created before signed-tag enforcement may not have equivalent evidence. ## Maintenance Review this assurance case when authentication, authorization, cryptography, deployment exposure, secret storage, release signing, or a trust boundary changes. Security reports may invalidate an argument and must be reflected here after remediation. # Release verification Observal release artifacts have GitHub keyless Sigstore provenance attestations. Release tags created after this policy was introduced are signed separately with gitsign. These checks prove different things: * Artifact verification binds a downloaded file and its SHA-256 digest to the Observal release workflow. * Tag verification binds a Git tag to the same GitHub Actions workflow identity. Historical tags may be unsigned. Never treat an artifact checksum alone as proof of origin. ## Download the release For a release produced by the release-branch pipeline, replace `v1.14.0` and `release/1.14` below with the exact version and minor line you want to verify: ```bash gh release download v1.14.0 --repo Observal/Observal --dir observal-release cd observal-release ``` ## Check the downloaded bytes The release includes `checksums.txt`: ```bash sha256sum --check checksums.txt ``` On macOS, use `shasum -a 256 --check checksums.txt`. ## Verify artifact provenance Install the [GitHub CLI](https://cli.github.com/) and verify each artifact you intend to run: ```bash gh attestation verify ./observal-linux-x64 \ --repo Observal/Observal \ --bundle ./build-provenance.intoto.jsonl \ --signer-workflow Observal/Observal/.github/workflows/release.yml@refs/heads/release/1.14 ``` Use the downloaded server archive or another CLI binary in place of `observal-linux-x64`. A successful result verifies the artifact digest, Sigstore certificate chain, source repository, and release workflow identity. GitHub keyless signing uses short-lived credentials issued to the workflow. Observal has no long-lived release private key stored on GitHub Releases or another download site. ## Verify a release tag Install [gitsign](https://github.com/sigstore/gitsign), clone the repository, and fetch the tag: ```bash git clone https://github.com/Observal/Observal.git cd Observal git fetch origin tag v1.14.0 gitsign verify-tag v1.14.0 \ --certificate-identity https://github.com/Observal/Observal/.github/workflows/release.yml@refs/heads/release/1.14 \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` The expected Fulcio certificate identity is the release workflow on the exact matching `refs/heads/release/X.Y` branch. The expected OIDC issuer is GitHub Actions. Verification also checks the signature's transparency-log evidence. A signed tag does not replace artifact provenance verification. Verify both when establishing the source commit and the exact downloaded bytes matters. Historical releases made before the release-branch migration used `refs/heads/main` as their signing identity. Verify those historical artifacts against that original identity explicitly. The new publishing workflow never falls back to the historical identity or accepts an unsigned tag. # ADR 0001: Agentic Resource Discovery **Status:** Accepted **Date:** 2026-09-20 **Deciders:** Observal maintainers ## Context Observal is install-first: a person builds an Agent from registry components, publishes it, and pulls it into a harness before a session starts. That model is right for repeatable, governed workflows and wrong for the moment a developer is mid-task and needs a capability that already exists in the registry. It also means the assistant only knows about resources someone installed ahead of time, so it rebuilds things that exist and tells users a capability is unavailable when it is one search away. The Agentic Resource Discovery specification (ARD) defines a common way to describe a resource and a common search API so any client can ask "what exists for this task?" across registries. It deliberately does **not** define installation, invocation, authentication, review, or telemetry. Observal already has all of those. The fit is: ARD says what exists, Observal says how to use it safely. This record fixes the decisions the discovery work depends on so later phases can be built against a stable target. The full plan lives outside the repo; this ADR is the part that must not drift. ## Decision 1: Pin the specification Observal implements **ARD v0.91 (Proposal)** as published at commit `b76f235a8f461876ad4f1e77abd0eb0eb302b48d` of [`ards-project/ard-spec`](https://github.com/ards-project/ard-spec) (2026-09-12). The JSON Schema, JSON-LD context, CDDL grammar, OpenAPI file and the official conformance CLI from that commit are vendored unmodified under `observal-server/vendor/ard/` (see `VENDORED.md` there for provenance and the update procedure). The spec is a proposal and will change. Observal tracks it by re-vendoring at a new commit, updating this ADR, and re-running the conformance tests, never by hand-editing vendored files or silently accepting whatever the live site says. ## Decision 2: Native models stay authoritative; ARD is a projection The existing `Agent`/`AgentVersion`, `McpListing`/`McpVersion`, `SkillListing`/`SkillVersion`, hook, prompt and sandbox tables remain the source of truth for installation, lifecycle, review and ownership. Discovery adds a read model, `discovery_entries`, populated from approved versions and refreshed on lifecycle events. Nothing that exists today is replaced or repurposed. The `/api/v1/agents` and component contracts are unchanged. ## Decision 3: Identifier scheme Every native resource gets exactly one permanent ARD identifier: ``` urn:air::: ``` * `` is the host of the deployment's public URL (`deployment.public_url` dynamic setting), which satisfies ARD's requirement that the publisher segment be a fully qualified domain name. * `` is one of `agent`, `mcp`, `skill`, `hook`, `prompt`, `sandbox` and occupies ARD's namespace segment. * `` is the native listing's primary key and occupies ARD's name segment. Example: `urn:air:observal.acme.com:skill:5f2c1a9e-8b1d-4d0c-9c4e-1f2a3b4c5d6e` **Why the UUID and not `namespace/slug`.** ARD requires the identifier to be a stable handle that survives the resource moving. Observal's `namespace/slug` changes on rename and on `transfer-owner`. The UUID never changes. The human-readable identity is still carried, as `obs:nativeRef` (`namespace/slug@version`), so clients can pivot to existing commands. **The publisher domain is pinned.** It is derived from `deployment.public_url` the first time a real fully qualified domain is seen and stored in the `discovery.publisher_domain` setting; after that, identifiers do not follow the public URL if the deployment moves hosts. Until a real domain is configured, identifiers use the placeholder `observal.local` and are expected to change once. Changing the pinned setting deliberately renames every identifier and is an administrator action. The artifact URL, by contrast, is a location and always follows the current public URL. Single-label hosts and IP addresses are never accepted as publishers. Externally ingested entries keep the identifier their publisher assigned. Observal never rewrites another publisher's URN. ## Decision 4: Endpoint base and well-known paths ``` GET /.well-known/ard.json manifest of public, approved entries GET /.well-known/ai-catalog.json predecessor path, same content POST /api/v1/ard/search ARD Search GET /api/v1/ard/agents ARD List (deterministic browse) POST /api/v1/ard/explore ARD Explore (facets), later phase GET /api/v1/ard/entries/{identifier} fetch one full entry (Observal-specific) GET /api/v1/artifacts/{kind}/{uuid}/{version} permanent versioned artifact with digest ``` The registry advertises `https:///api/v1/ard` as its operational base through an entry of type `application/ai-registry+json` in the manifest. ARD defines no operation for fetching a complete entry by identifier; clients are expected to fetch the artifact `url`. Observal adds `/entries/{identifier}` for its own CLI and UI. It is not part of the conformance surface. ## Decision 5: `federation` when omitted The spec's default for the `federation` field is `auto`. Observal honours that literally: an omitted field is treated as `auto`, **bounded by the administrator's upstream allowlist**. With the default empty allowlist this is functionally local-only. When an admin allows upstreams, `auto` and `referrals` only ever reach those. Rejected alternative: default to `none`. That would make a conformant client receive different behaviour than the spec promises, and the privacy goal is achieved just as well by an empty allowlist. Search text may contain sensitive information (incident details, unreleased names, internal service names). It never leaves the deployment unless an admin has allowed a specific upstream, and results from upstreams always carry their `source`. ## Decision 6: Visibility Discovery narrows the signed-in user's registry view to their own namespace and teamspaces they currently belong to. This is intentionally stricter than registry browsing/install for public resources in other namespaces: a public resource can be installed by reference without appearing in a user's ARD search. The same scope applies to search, list, inspect, and versioned artifact reads, including imported A2A agents. Membership is checked at query time, so removing a member revokes their ARD access to that team's public and private entries. Submitter status alone does not grant access to an old teamspace. Two filters apply, in order: 1. **Who can see the listing** — ordinary signed-in users see personal listings they own (public or private) and listings in *all* teamspaces where they are current members (public or private). Global reviewers retain their public review scope; admins and super-admins still see everything. Anonymous calls retain the public-catalog behavior described below. 2. **Which lifecycle states are returned** — approved versions to everyone who passes filter 1; pending and rejected versions only to users whose effective component permission is `owner` (the same rule install uses for owner fallback); archived entries excluded unless the query filters on `obs:lifecycle` explicitly. Unauthenticated requests receive only public, approved entries, and only when the registry-wide `deployment.public_registry_enabled` setting is on (default off), the same switch that governs anonymous browsing elsewhere. When it is off, unauthenticated search returns an empty result set and the well-known manifest carries only the registry's own `application/ai-registry+json` entry, both of which are conformant. The well-known manifest is always computed as an unauthenticated request. ## Decision 7: Media types and compatibility inputs Emitted (canonical) types: | Observal resource | ARD `type` | | ----------------- | ---------------------------------------------------------------------- | | Skill | `application/ai-skill+md` | | MCP server | `application/mcp-server-card+json` (a descriptor, not the live stream) | | Prompt | `application/vnd.observal.prompt+json` | | Sandbox | `application/vnd.observal.sandbox+json` | | Hook | `application/vnd.observal.hook+json` | | Agent | `application/vnd.observal.agent+json` | | External, unknown | preserved exactly as published | An Observal Agent is an installable configuration, not a running service, so it is never emitted as `application/a2a-agent-card+json`. (ADR 0002 adds remote A2A agents as imported entries of that type, and delegation to both kinds.) Accepted and normalised on input, with the original preserved in `raw_entry`: | Input | Normalised to | | ------------------------------------------------------- | ----------------------------------------------- | | `urn:ai:…` | `urn:air:…` | | `application/mcp-server`, `application/mcp-server+json` | `application/mcp-server-card+json` | | `application/ai-skill` | `application/ai-skill+md` | | `/.well-known/ai-catalog.json`, `rel="ai-catalog"` | treated as `/.well-known/ard.json`, `rel="ard"` | ## Decision 8: Relevance is the only thing in `score` ARD's `score` (0–100) is semantic relevance and nothing else. Approval, publisher verification, liveness, harness compatibility and adoption are returned as separate, named fields (`obs:approval`, `obs:trust`, `obs:availability`, `obs:supportedHarnesses`, …) and are never blended into `score`. Policy eligibility is applied as a filter before ranking, not as a weight inside it. ## Decision 9: What is deliberately deferred * **Web ingestion.** ARD §5.2 says every registry MUST crawl entry sources. Observal defers this to the external-sources phase. The vendored conformance tool passes without it; until that phase ships, Observal is an ARD *publisher* and *search endpoint*, not a complete ARD *registry*, and says so here rather than claiming otherwise. * **Explore.** Optional in the spec; added with search-quality work. * **Trust manifest verification.** Entries Observal publishes carry a `trustManifest.identity` of `https://` (an HTTPS FQDN URI string, `identityType: "https"`) bound to the URN's publisher segment. Verification of third-party manifests arrives with external ingestion. ## Decision 10: Search baseline needs no external service The first search implementation is PostgreSQL full-text search plus `pg_trgm` (already enabled). It must remain a complete, supported configuration. Dense retrieval is an optional later addition behind a provider interface and an explicit infrastructure change (pgvector); its absence never disables discovery. ## Consequences * Discovery reaches all ten harnesses through the bundled skill and the CLI, which already ship to every harness on login. No new runtime is required for the first milestone. * Per-harness behaviour (how MCP is installed, whether tools refresh mid-session, whether a hook can inject context) is recorded as verified facts in the shared harness registry and tested, so later phases gate on data rather than assumptions. * Anything that permanently changes a developer's setup, or that can write or delete, keeps asking first. Approved skills and prompts, being text, may be loaded automatically. * Conformance is tested in CI against the vendored tool at the pinned commit. A spec change is a deliberate re-vendor, reviewed like any other dependency update. # Contributing Guide Thank you for considering contributing to Observal. Contributions of all kinds are welcome: bug reports, bug fixes, new features, documentation improvements, and tests. > \[!TIP] This page is a quick-start summary. For the full setup walkthrough, architecture notes, and detailed workflows, see the [Development Guide](/contributing/development_guide). For new Python tests, follow the [Testing Guide](/contributing/testing_guide). Reviewers and maintainers must follow the [Code Review Standard](/contributing/code-review). > \[!IMPORTANT] **Discord is our primary communication channel.** Join at [discord.observal.io](https://discord.observal.io) and ask questions in **#contributing**, report bugs in **#bug**, or discuss ideas in **#feature-requests**. GitHub issues and PRs are for concrete, actionable items, not exploratory discussion. Please read our [Code of Conduct](https://github.com/Observal/Observal/blob/main/CODE_OF_CONDUCT.md) and [AI Policy](/contributing/ai_policy) before contributing. Interested in how contributors are paid? See [Contributor Payments](https://github.com/BlazeUp-AI/Observal/tree/main/docs/contributor-payments.md). > Parts of this guide were inspired by the contributing documentation from [AnkiDroid/Anki-Android](https://github.com/ankidroid/Anki-Android). They set a great standard for OSS contributor docs and were one of the first open-source projects some of our maintainers were part of. If you are looking for another welcoming OSS project, check them out. *** ## Table of Contents * [Getting Started](#getting-started) * [Finding Work](#finding-work) * [Making Changes](#making-changes) * [Submitting a Pull Request](#submitting-a-pull-request) * [Code Review Standard](/contributing/code-review) * [Reporting Issues](#reporting-issues) * [License](#license) * [CLA](#contributor-license-agreement-cla) *** ## Getting Started ### Prerequisites * Docker and Docker Compose * [uv](https://docs.astral.sh/uv/) (Python 3.11+) * Node.js 22.6+ and pnpm (for the web frontend and `make test`; matches `package.json` `engines` and `--experimental-strip-types`) * Git ### Fork and Clone ```bash git clone https://github.com/YOUR-USERNAME/Observal.git cd Observal git remote add upstream https://github.com/Observal/Observal.git ``` ### Running Locally No configuration needed for local development. All settings have working defaults. **Full stack (Docker):** ```bash cp .env.example .env make rebuild ``` `make rebuild` (also available as `make rebuild-fast`, the same target) is the one command for backend, frontend, dependency, schema, migration, ClickHouse setup, init path, worker, and Compose topology changes. It builds the shared API image once (reused by the API, init, and worker services), builds the web image, then recreates whatever changed. Wait for services to be healthy, then: ```bash uv tool install --editable . observal auth login ``` The stack starts at `http://localhost` (nginx LB on port 80). The `.env.example` seeds demo accounts on first startup, log in with `super@demo.example` / `super-changeme` for admin access. See [SETUP.md](https://github.com/Observal/Observal/blob/main/SETUP.md) for all credentials. **Frontend only:** ```bash cd web && pnpm install && pnpm dev ``` Set `NEXT_PUBLIC_API_URL=http://localhost` in `web/.env.local` if the backend is on a different host. > \[!NOTE] See the [Development Guide](/contributing/development_guide) for the full environment setup and troubleshooting steps. *** ## Finding Work Check [open issues](https://github.com/Observal/Observal/issues) before starting. Look for **good first issue** if you are new. For larger changes, open an issue or discuss in **#contributing** on Discord before writing code. ### Claiming Issues * **`/take`** on any `good first issue` or `help wanted` issue to self-assign. * **`/drop`** to release an issue you can no longer work on. * Max **2 open assigned issues** at a time. * Issues with no activity for **30 days** are automatically unassigned. > \[!WARNING] Issues labeled `keep open` cannot be claimed. Anyone may submit a PR for those without assignment. *** ## Making Changes ### Branch Naming ``` feature/skill-registry fix/clickhouse-insert-timeout docs/update-setup-guide ``` Never commit directly to `main` or a protected `release/X.Y` branch. Normal changes target `main`. Release preparation and backport PRs target their matching `release/X.Y` line. Never merge a release line back into `main`. See the [release guide](https://github.com/BlazeUp-AI/Observal/tree/main/docs/self-hosting/releasing.md) for beta, RC, stable, and backport workflows. ### Code Style ```bash make hooks # install pre-commit hooks (do this first) make format # auto-format Python and TypeScript make lint # run all linters ``` Python is formatted with `ruff`. Dockerfiles with `hadolint`. Pre-commit hooks enforce both. ### SPDX Headers Every source file needs SPDX headers. The pre-commit hook adds them automatically. ```python # SPDX-FileCopyrightText: 2026 Your Name # SPDX-License-Identifier: Apache-2.0 ``` Use `//` for TypeScript, `` for Markdown. CI will block merge if any file is missing headers. ### Testing ```bash make test # quick make test-v # verbose ``` All tests must pass before submitting. Tests mock all external services so Docker is not required. Include tests for any feature or bug fix. New Python tests should follow the [Testing Guide](/contributing/testing_guide). In short, keep tests hermetic, assert behavior over implementation details, use small local helpers for setup, and avoid touching real user configuration. ### Commit Messages Follow [Conventional Commits](https://www.conventionalcommits.org/): ``` feat(cli): add skill submit command fix(telemetry): handle null span timestamps docs: update contributing guide ``` Subject line under 72 characters, imperative mood, no trailing period. ### Changelog Add an entry under `[Unreleased]` in [CHANGELOG.md](https://github.com/Observal/Observal/blob/main/CHANGELOG.md) for any user-facing change. *** ## Submitting a Pull Request > \[!IMPORTANT] Read the [AI Policy](/contributing/ai_policy) before submitting. AI-assisted contributions are welcome but must meet the standards described there. **Autonomous coding agents (Devin, SWE-agent, OpenHands, and similar tools that write and submit code without meaningful human authorship) are not permitted**, see the AI Policy for the legal and practical reasons. PRs that show obvious signs of unreviewed AI output will be closed without review. 1. Rebase against `main` before opening: ```bash git fetch upstream && git rebase upstream/main ``` 2. Push your branch and open a PR against `main`. 3. Fill in the PR template completely. PRs with unfilled or placeholder sections will be closed. 4. Ensure CI passes (linters, tests, docker build). 5. Add a changelog entry if your change is user-facing. 6. Respond to review feedback promptly. Keep PRs focused on a single concern. Smaller PRs are easier to review and faster to merge. All pull requests are evaluated under the [Code Review Standard](/contributing/code-review). It defines reviewer responsibilities, required approvals, review freshness, merge gates, and the conditions that require changes or rejection. *** ## Reporting Issues ### Bugs Search [existing issues](https://github.com/Observal/Observal/issues) first. Include: * Steps to reproduce * Expected vs actual behaviour * OS, Python, Node.js, Docker versions * Error logs or screenshots ### Feature Requests Describe the problem you are solving, not just the solution. Discuss in **#feature-requests** on Discord first for larger features. *** ## License All code is licensed under [Apache-2.0](https://github.com/BlazeUp-AI/Observal/tree/main/LICENSE/README.md). *** ## Contributor License Agreement (CLA) The [CLA-assistant](https://cla-assistant.io) bot will prompt you to sign the [Observal CLA](https://github.com/Observal/Observal/blob/main/CLA.md) on your first PR. You only need to sign once. For corporate contributions, contact . # Code Review Standard Code review protects Observal users, contributors, and maintainers. Its purpose is not merely to make a pull request pass CI. Reviewers are responsible for deciding whether a change is correct, secure, understandable, appropriately tested, and safe to maintain. This standard applies to every pull request merged into a protected branch. The depth of review should match the risk of the change, but the approval and merge requirements apply even to small changes. ## Required reviewers Every pull request requires: * Approval from at least two eligible reviewers. An eligible reviewer is a project collaborator whose approval GitHub counts toward the protected branch. * Approval from a code owner when the changed paths have an applicable entry in `CODEOWNERS`. A code-owner approval may count as one of the two required approvals. * Additional review from someone qualified in any area that the assigned reviewers cannot confidently assess, such as authentication, authorization, cryptography, privacy, database migrations, release infrastructure, accessibility, or concurrency. Anyone may comment and provide useful review feedback. Reviewers who are not eligible approvers should state that their review is advisory. A reviewer covering only part of a pull request must identify the files or concerns they reviewed and must not imply approval of the remainder. Author self-approval is prohibited. An approval from a co-author or anyone who contributed commits to the pull request does not count toward the two independent approvals. ## Reviewer procedure ### 1. Understand the change Before reviewing individual lines: 1. Read the pull request description, linked issue, prior design discussion, and testing evidence. 2. Confirm that the proposed behavior is wanted and that the pull request addresses the stated problem. 3. Check that the change is focused. Unrelated refactors, formatting, generated files, or dependency updates should be separated unless they are necessary for the change. 4. Identify high-risk areas and request a qualified reviewer early when needed. If the purpose, expected behavior, or risk is unclear, ask for clarification before approving. ### 2. Review the complete change Review every human-written line in the diff and enough surrounding code to understand its effect. Do not rely only on the changed lines. Trace callers, data flow, failure paths, configuration, migrations, and user-visible behavior where relevant. Generated files may be sampled when their source and generation command are clear. Lockfiles still require review for unexpected packages, sources, versions, and integrity data. Automated checks support human review but do not replace it. A green pipeline cannot detect every authorization error, privacy leak, race, incompatible interface, or incorrect product decision. ### 3. Validate the evidence Confirm that the pull request explains how the change was tested. Run focused tests or reproduce the behavior when the risk, novelty, or user impact warrants independent validation. Ask for screenshots, recordings, logs, benchmarks, migration output, or a demonstration when code inspection alone is insufficient. A reviewer must not approve code they do not understand. Ask questions or bring in another reviewer instead. ### 4. Give actionable feedback Use **Request changes** for blocking findings. Explain the risk and the condition that must be satisfied. Mark optional improvements clearly as non-blocking or as a nit. Prefer comments about correctness, risk, maintainability, and documented project conventions over personal style preferences. Suggest a follow-up issue only when the current change is safe and complete without the follow-up. The author must acknowledge every blocking thread. A thread may be resolved when the change is made or when the author and reviewer agree, with the reasoning recorded in the pull request. ### 5. Review the latest revision Review the final diff, not only the first version submitted. After a material push, prior approvals are stale. All required approvers must review the delta and approve or explicitly reaffirm approval on the latest material revision. A push is material when it changes behavior, interfaces, security or privacy properties, data handling, dependencies, migrations, build or release logic, test strategy, or the agreed scope. Typographical and other clearly non-behavioral corrections are not normally material, but a reviewer may request another review whenever the impact is uncertain. ## What every review must cover ### Correctness and design Confirm that: * The implementation satisfies the documented requirements and preserves relevant invariants. * Error handling is explicit and does not hide failed operations, partial writes, or data loss. * Edge cases, empty and malformed inputs, retries, concurrency, idempotency, and resource cleanup are handled where relevant. * The design follows existing project architecture and uses an existing helper or platform feature where appropriate. * The change does not introduce unnecessary abstraction, configuration, dependencies, dead code, or speculative flexibility. * Performance, availability, and operational costs are reasonable for expected production use. ### Security and privacy Treat all external input and cross-tenant data as untrusted. Confirm that: * Authentication and authorization are enforced at the server boundary, not only in the user interface. * Access checks cover object ownership, team boundaries, administrative roles, and indirect lookup paths. * Inputs are validated and are safe from injection, path traversal, request forgery, unsafe redirects, and insecure deserialization. * Secrets, tokens, credentials, personal data, and telemetry are not exposed through logs, errors, URLs, browser storage, analytics, or generated artifacts. * Data collection, retention, export, deletion, and redaction remain consistent with the documented privacy model. * Cryptography and security controls use established project mechanisms and secure defaults. * New dependencies, actions, container images, and downloaded tools are necessary, integrity-pinned where supported, and from trustworthy sources. Do not discuss an undisclosed vulnerability in a public pull request. Follow the private process in the [Security Policy](https://github.com/BlazeUp-AI/Observal/tree/main/SECURITY.md). ### Tests and CI Confirm that: * New behavior has tests at the smallest effective level, with integration or end-to-end coverage where boundaries or user workflows require it. * A bug fix includes a regression test that fails without the fix whenever practical. * Tests assert observable behavior, exercise failure paths, and would detect a real regression. * Tests are deterministic, isolated from real user data and external services, and no more complex than necessary. * Required CI checks pass for the final material revision. Flaky, skipped, cancelled, or unrelated failures must be investigated and documented, not silently ignored. ### Compatibility and data safety Confirm that: * Public APIs, CLI commands, configuration, stored data, deployment files, supported platforms, and harness integrations remain compatible unless an intentional change is approved and documented. * Breaking changes include a clear rationale, user migration path, release note, and coordinated updates to every in-repository caller. * Database migrations use the repository migration tooling, have been inspected, preserve existing data, and have a safe upgrade and recovery plan. * Changes that cross versions, workers, caches, queues, or rolling deployments account for mixed-version operation where applicable. ### Documentation and user experience Confirm that: * User-visible behavior, APIs, commands, configuration, setup, and operational procedures are documented in the same pull request. * CLI syntax changes are reflected in bundled skill documentation. * User-facing changes include the required changelog entry and accurate screenshots or other evidence when applicable. * Interfaces remain accessible, understandable, responsive, and consistent with established design patterns. * Comments explain non-obvious intent and trade-offs rather than restating the code. ### Licensing and provenance Confirm that: * New files have the correct SPDX copyright and license headers. * The contributor has satisfied the CLA and followed the [AI Policy](/contributing/ai_policy). * Copied or adapted material is compatible with Apache-2.0, properly attributed, and accompanied by any required notices. * New dependencies, assets, generated code, and vendored files have acceptable licenses and documented origins. ### Scope and maintainability Confirm that: * The pull request contains one coherent change and no unrelated cleanup. * The implementation is readable by maintainers who did not author it. * Temporary code, debugging output, compatibility workarounds, and obsolete paths are removed unless explicitly justified. * The pull request description, commit history, and documentation accurately describe the final implementation. ## Approval and merge requirements A pull request is acceptable for merge only when all of the following are true: * Two eligible reviewers have approved the latest material revision. * A code owner has approved where `CODEOWNERS` applies. * No reviewer has an outstanding **Request changes** review. * Every review thread is resolved with its outcome recorded. * All required CI checks pass on the mergeable revision. * Required tests, documentation, changelog entries, screenshots, migration evidence, SPDX headers, and license information are present. * The CLA and other repository policy checks pass. * The pull request is focused, understandable, and ready to operate without relying on an unspecified follow-up. The person merging the pull request performs a final gate check. Merge permission is not permission to bypass this standard. ## When to request changes or reject a pull request A reviewer must withhold approval and request changes when any blocking issue remains, including: * Incorrect behavior, an unhandled failure mode, data-loss risk, or an unresolved security or privacy concern. * Missing or ineffective tests for behavior that can reasonably be tested. * A breaking or operationally unsafe change without an approved migration and recovery plan. * Missing required documentation, licensing information, attribution, or policy compliance. * Scope or complexity that prevents a reliable review. The reviewer may require the pull request to be split or redesigned. * A material revision that has not been reviewed by the required approvers. A pull request should be rejected or closed when it: * Conflicts with an established project decision or does not solve an accepted project need. * Duplicates existing work without a reason to replace it. * Introduces malicious, deceptive, obfuscated, or unverifiable behavior. * Contains code or assets that the contributor cannot legally license to the project. * Violates the Code of Conduct, CLA, AI Policy, responsible disclosure process, or other repository policy. * Remains abandoned after maintainers have clearly requested the information or changes needed to continue. Reviewers should explain rejection clearly and respectfully. Rejection is a decision about the proposed change, not the contributor. ## Review conduct and disagreements Follow the [Code of Conduct](https://github.com/BlazeUp-AI/Observal/tree/main/CODE_OF_CONDUCT.md). Be specific, constructive, and proportionate to risk. Authors should not interpret questions as approval, and reviewers should not use approval as leverage for unrelated preferences. Resolve disagreements using documented requirements, reproducible evidence, and project architecture. If consensus cannot be reached, ask another domain expert or maintainer to decide and record the decision in the pull request. Do not leave a pull request indefinitely blocked by an unstated concern. Urgent and embargoed security changes may be reviewed privately by a restricted group, but independent review, testing, and approval are not waived. ## Further reading This standard was informed by established review practices from: * [Google Engineering Practices: What to look for in a code review](https://google.github.io/eng-practices/review/reviewer/looking-for.html) * [GitLab Code Review Guidelines](https://docs.gitlab.com/development/code_review/) * [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) * [Django Contribution Checklist](https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/submitting-patches/#contribution-checklist) * [OpenSSF Source Code Management Platform Configuration Best Practices](https://best.openssf.org/SCM-BestPractices/) # Development Guide > \[!IMPORTANT] **Discord is the primary place to ask questions.** Join at [discord.observal.io](https://discord.observal.io). > > * **#contributing**, setup help, workflow questions, anything about the contribution process > * **#bug**, discuss bugs before filing a GitHub issue > * **#feature-requests**, pitch ideas and get early feedback before writing code > > GitHub issues and PRs are for concrete, actionable items. Exploratory discussion belongs on Discord first. > Parts of this guide structure were inspired by the [AnkiDroid Development Guide](https://github.com/ankidroid/Anki-Android/wiki/Development-Guide). Attribution given with thanks, check them out if you want another welcoming OSS project to contribute to. *** ## Table of Contents * [Community Standards](#community-standards) * [Prerequisites](#prerequisites) * [Git Setup](#git-setup) * [SSH authentication](#ssh-authentication) * [Git identity](#git-identity) * [Fork and clone](#fork-and-clone) * [First-time Setup](#first-time-setup) * [Install pre-commit hooks](#install-pre-commit-hooks) * [Start the Docker stack](#start-the-docker-stack) * [Install the CLI](#install-the-cli) * [Demo accounts](#demo-accounts) * [Make Targets Reference](#make-targets-reference) * [Architecture Overview](#architecture-overview) * [Git Workflow](#git-workflow) * [Making a new branch](#making-a-new-branch) * [Keeping your branch up to date](#keeping-your-branch-up-to-date) * [Dealing with merge conflicts](#dealing-with-merge-conflicts) * [Submitting a pull request](#submitting-a-pull-request) * [After review feedback](#after-review-feedback) * [Working on the Backend](#working-on-the-backend) * [Running tests](#running-tests) * [Testing conventions](#testing-conventions) * [Running a single test](#running-a-single-test) * [Code coverage](#code-coverage) * [Adding a database migration](#adding-a-database-migration) * [Connecting to PostgreSQL directly](#connecting-to-postgresql-directly) * [Connecting to ClickHouse directly](#connecting-to-clickhouse-directly) * [Debugging the API](#debugging-the-api) * [Working on the Frontend](#working-on-the-frontend) * [Running the frontend in isolation](#running-the-frontend-in-isolation) * [Design system](#design-system) * [Adding a new API endpoint to the frontend](#adding-a-new-api-endpoint-to-the-frontend) * [Screenshots for UI changes](#screenshots-for-ui-changes) * [Working on the CLI](#working-on-the-cli) * [Reinstalling after changes](#reinstalling-after-changes) * [Harness Recommendations](#harness-recommendations) * [VS Code](#vs-code) * [PyCharm / IntelliJ](#pycharm--intellij) * [Pre-commit Hooks](#pre-commit-hooks) * [Pull Request Checklist](#pull-request-checklist) * [Getting Help](#getting-help) *** ## Community Standards We are a small, active community and we take the quality of interactions seriously. > \[!WARNING] The following will result in a **moderator warning**. A second violation results in a **temporary or permanent ban** from the repository and Discord: > > * Pinging contributors, maintainers, or reviewers unnecessarily (outside of a direct reply on your own open PR or issue) > * Submitting low-effort or unreviewed PRs (slop), including unreviewed AI output or autonomous agent submissions > * Violating the [Code of Conduct](https://github.com/Observal/Observal/blob/main/CODE_OF_CONDUCT.md) in any channel > * Harassing reviewers over merge timelines Maintainers volunteer their time. Treat them accordingly. ### A note on autonomous coding agents **Autonomous coding agents (Devin, SWE-agent, OpenHands, etc.) are not permitted to submit PRs.** This is a legal constraint, not a quality judgment. The US Copyright Office's 2025 guidance confirms that purely AI-generated code has no copyright owner, which breaks our CLA and our ability to license and enforce the project. See the [AI Policy](/contributing/ai_policy) for the full explanation. *** ## Prerequisites | Tool | Version | Install | | ------- | ------------------- | ------------------------------------------------------------------------- | | Docker | 24+ with Compose v2 | [docs.docker.com](https://docs.docker.com/get-docker/) | | uv | latest | `curl -LsSf https://astral.sh/uv/install.sh \| sh` | | Node.js | 22.6+ | Via [nvm](https://github.com/nvm-sh/nvm) or [mise](https://mise.jdx.dev/) | | pnpm | 10+ | `npm install -g pnpm` | | Git | 2.28+ | [git-scm.com](https://git-scm.com/) | Verify your Docker installation supports Compose v2: ```bash docker compose version # should print: Docker Compose version v2.x.x ``` If you see `docker-compose: command not found` or a v1 version, upgrade Docker Desktop or install the Compose plugin separately. *** ## Git Setup ### SSH authentication Using SSH is strongly recommended, it avoids password prompts and is required if you enable 2FA on GitHub. ```bash # Generate a key (skip if you already have one at ~/.ssh/id_ed25519) ssh-keygen -t ed25519 -C "your@email.com" # Copy the public key to your clipboard cat ~/.ssh/id_ed25519.pub ``` Add the output to **GitHub → Settings → SSH and GPG keys → New SSH key**. Test it: ```bash ssh -T git@github.com # Hi your-username! You've successfully authenticated... ``` ### Git identity Set your identity so it matches your GitHub account: ```bash git config --global user.name "Your Name" git config --global user.email "your@email.com" ``` If you want to sign commits with GPG as well (optional but appreciated): ```bash git config --global commit.gpgsign true git config --global user.signingkey YOUR_GPG_KEY_ID ``` ### Fork and clone You only need to do this once. For every subsequent contribution, just create a new branch on your existing fork. 1. Fork the repository on GitHub (top-right **Fork** button). 2. Clone your fork using SSH: ```bash git clone git@github.com:YOUR-USERNAME/Observal.git cd Observal ``` 3. Add the upstream remote so you can pull in changes from the main repo: ```bash git remote add upstream https://github.com/Observal/Observal.git ``` 4. Verify your remotes: ```bash git remote -v # origin git@github.com:YOUR-USERNAME/Observal.git (fetch) # origin git@github.com:YOUR-USERNAME/Observal.git (push) # upstream https://github.com/Observal/Observal.git (fetch) # upstream https://github.com/Observal/Observal.git (push) ``` *** ## First-time Setup ### Install pre-commit hooks Do this **before your first commit**. The hooks run ruff, format checkers, secret scanners, SPDX header injection, and migration chain validation automatically. ```bash make hooks ``` ### Start the Docker stack ```bash cp .env.example .env make up ``` The first build takes several minutes while Docker downloads and compiles images. Subsequent starts are fast. Watch progress with: ```bash make logs ``` For code, dependency, and Compose changes, use: ```bash make rebuild ``` `make rebuild-fast` is the same target under another name. It is not a separate dev profile and does not enable hot reload. It builds only the two app images that contain local code: * `observal-api`, shared by the API, init, and worker services * `observal-web`, used by the web service Then it starts the full stack with the freshly built images. Wait until all services are healthy: ```bash docker compose -f docker/docker-compose.yml ps # All services should show "healthy" or "running" ``` **Default endpoints:** | Service | URL | | -------------------- | ----------------------- | | LB (all traffic) | `http://localhost` | | Web UI (direct) | `http://localhost:3000` | | Prometheus, optional | `http://localhost:9090` | | Grafana, optional | `http://localhost:3001` | | ClickHouse HTTP | `http://localhost:8123` | ### Install the CLI ```bash uv tool install --editable . ``` ### Log in ```bash observal auth login ``` On a fresh server, this bootstraps the admin account automatically. Use `super@demo.example` / `super-changeme` for the super\_admin account. ### Demo accounts Seeded automatically on first startup: | Email | Password | Role | | -------------------- | ---------------- | ------------ | | `super@demo.example` | `super-changeme` | super\_admin | | `admin@demo.example` | `admin-changeme` | admin | | `dev@demo.example` | `dev-changeme` | developer | | `user@demo.example` | `user-changeme` | user | *** ## Make Targets Reference ```bash make up # start the full Docker stack make down # stop the stack make rebuild # rebuild app images and restart (alias: make rebuild-fast) make logs # tail logs from all services make test # run the test suite quickly make test-v # verbose test output make lint # ruff check + hadolint make format # ruff format (auto-fix) make check # pre-commit on all files make hooks # install pre-commit hooks make reset # nuke all volumes and rebuild from scratch (destructive) ``` ### Which rebuild target should I use? | Situation | Target | Notes | | ------------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | Any code, dependency, schema, init, worker, or Compose change | `make rebuild` | `make rebuild-fast` is the same target. Rebuilds the shared API image (API, init, worker) and the web image, then recreates changed services. | | Cache looks stale or the stack behaves unexpectedly | `make rebuild-clean` | Rebuilds from scratch without the Docker cache and removes volumes. | `make down` stops containers started by `make rebuild`. *** ## Architecture Overview Observal is a monorepo: ``` observal-server/ FastAPI backend (Python) observal_cli/ CLI and session exporters (Python) web/ Next.js 16 / React 19 frontend (TypeScript) tests/ Shared test suite (~1500 tests, 96 files) docker/ Docker Compose and Dockerfiles docs/ Documentation scripts/ Dev tooling scripts ``` **Databases:** * **PostgreSQL**, relational data (users, agents, registry, feedback) * **ClickHouse**, session events, aggregates, audit events, and security events They are not interchangeable. Never write telemetry to Postgres or relational data to ClickHouse. **Supporting services:** Redis (pub/sub + arq job queue), arq worker, nginx reverse proxy. Prometheus and Grafana are optional. See [AGENTS.md](https://github.com/Observal/Observal/blob/main/AGENTS.md) for a complete map of every important file and service. *** ## Git Workflow ### Making a new branch Always branch from the latest `main`. Never commit directly to `main`. ```bash git checkout main git pull upstream main git checkout -b feature/my-feature # or: fix/my-bug, docs/my-doc ``` ### Keeping your branch up to date If `main` has moved forward while you were working, rebase your branch onto it: ```bash git checkout main git pull upstream main git checkout feature/my-feature git rebase main ``` Rebasing keeps history linear and makes your PR easier to review than a merge commit would. ### Dealing with merge conflicts Conflicts arise when the same file was changed in both `main` and your branch. During `git rebase main`, git will pause and show you: ``` CONFLICT (content): Merge conflict in some/file.py ``` Open the file and look for conflict markers: ```python <<<<<<< HEAD # this is what's on main ======= # this is what's in your branch >>>>>>> feature/my-feature ``` Edit the file to the correct final state (keeping whichever changes are right, or combining them), then: ```bash git add some/file.py git rebase --continue ``` If you get confused and want to start over: ```bash git rebase --abort ``` After resolving and pushing, use `--force-with-lease` rather than `--force`, it's safer: ```bash git push origin feature/my-feature --force-with-lease ``` ### Submitting a pull request 1. Make sure your branch is rebased on the latest `main` (see above). 2. Push to your fork: ```bash git push origin feature/my-feature ``` 3. GitHub will show a banner on your fork offering to open a PR. Click it, or go to the [Observal repository](https://github.com/Observal/Observal) directly. 4. Fill in the PR template completely. PRs that do not follow the template will be closed. 5. Link the related issue if one exists (`Fixes #123` in the PR body closes it automatically on merge). ### After review feedback If a reviewer requests changes, make them on the same branch and push again. Do not open a new PR. If your changes are small fixes to an existing commit, amend rather than adding new commits: ```bash git add the-changed-file.py git commit --amend --no-edit git push origin feature/my-feature --force-with-lease ``` *** ## Working on the Backend ### Running tests Tests mock all external services. Docker does not need to be running. ```bash make test make test-v make test-eval-completeness make test-adversarial make test-all # or directly: cd observal-server uv run --with pytest --with pytest-asyncio --with hypothesis --with pyarrow pytest ../tests/ -q ``` ### Testing conventions New Python tests should follow the [Testing Guide](/contributing/testing_guide). The current suite has mixed historical patterns, so do not rewrite old tests only for style. When adding or touching tests, prefer the clean pattern documented there: one behavior area per file, small local helper factories, hermetic CLI and API test setup, explicit async mocks, and behavior-focused assertions. ### Running a single test ```bash cd observal-server uv run pytest ../tests/test_registry_types.py -q # or a specific test function: uv run pytest ../tests/test_registry_types.py::TestMcpRoutes::test_submit_mcp -v ``` ### Code coverage Coverage is collected automatically on the `3.13` matrix run in CI. To generate it locally: ```bash cd observal-server uv run pytest ../tests/ --cov=../observal_cli --cov=. --cov-report=html -q # open htmlcov/index.html in your browser ``` ### Adding a database migration ```bash bash scripts/new_migration.sh "describe_your_change" ``` Edit the generated file in `observal-server/alembic/versions/`. Then verify the chain is intact: ```bash python3 scripts/check_migrations.py ``` > \[!CAUTION] Never edit an existing migration file. Always create a new one. A broken migration chain blocks CI and prevents the server from starting. Apply migrations to your local stack: ```bash docker compose -f docker/docker-compose.yml run --rm init ``` ### Connecting to PostgreSQL directly ```bash docker compose -f docker/docker-compose.yml exec db \ psql -U observal -d observal ``` Useful for inspecting tables, running manual queries, or verifying migration results. ### Connecting to ClickHouse directly ClickHouse exposes an HTTP interface. Use the Play UI in your browser: ``` http://localhost:8123/play ``` Or query from the terminal: ```bash curl -s "http://localhost:8123/?query=SELECT+count()+FROM+session_events" \ -u "default:" ``` Or connect with the CLI client: ```bash docker compose -f docker/docker-compose.yml exec clickhouse \ clickhouse-client --user default ``` ### Debugging the API The API server runs with `--reload` in development mode. Log output is available via: ```bash docker compose -f docker/docker-compose.yml logs -f observal-api ``` To add a breakpoint, use `breakpoint()` in Python code, the debugger will pause in the container's stdout. Or attach a remote debugger by adding `debugpy` to your dev dependencies and exposing a debug port. The OpenAPI docs are available at (API port, not through the LB which blocks these paths): * `http://localhost:8000/docs` (Swagger UI, requires direct API port access) * `http://localhost:8000/redoc` (ReDoc, requires direct API port access) > Note: The nginx LB blocks `/docs`, `/redoc`, and `/openapi.json` in production. For local dev, expose the API port directly or use `docker compose exec`. *** ## Working on the Frontend ### Running the frontend in isolation ```bash cd web pnpm install pnpm dev ``` Create `web/.env.local` with: ```bash NEXT_PUBLIC_API_URL=http://localhost ``` The frontend proxies all `/api/v1/*` calls to the backend URL set by `NEXT_PUBLIC_API_URL`. ### Design system | Token | Value | | ------------- | ----------------------------------------------- | | Color space | OKLCH | | Themes | `light`, `dark`, `midnight`, `forest`, `sunset` | | Display font | Archivo | | Body font | Albert Sans | | Code font | JetBrains Mono | | Spacing base | 4pt | | Components | shadcn/ui | | Charts | Recharts | | Data fetching | TanStack Query | | Tables | TanStack Table | All themes are defined in `web/src/app/globals.css` as CSS custom properties. Semantic tokens (`background`, `foreground`, `card`, `border`, `primary`, `secondary`, `accent`, `destructive`, `success`, `warning`, `info`) map to OKLCH values per theme. When adding new UI, use the semantic tokens, never hardcode colors. Check all five themes look correct before submitting. ### Adding a new API endpoint to the frontend 1. Add the TypeScript response type to `web/src/lib/types.ts`. 2. Add the fetch call to `web/src/lib/api.ts`. 3. Add the TanStack Query hook to `web/src/hooks/use-api.ts`. 4. Use the hook in your component. ### Screenshots for UI changes > \[!IMPORTANT] Any PR that touches the web frontend must include **screenshots of all affected screens** in the PR description. This is required regardless of how small the change is. Attach screenshots directly to the PR body, not as review comments. *** ## Working on the CLI ### Reinstalling after changes The CLI is installed as a uv tool. Changes to source files are reflected immediately because it is installed in editable mode (`--editable`). If you add new entry points or change `pyproject.toml`, reinstall: ```bash uv tool install --editable . --reinstall ``` ### Testing session delivery Run a harness session with hooks installed, then reconcile and inspect exporter status: ```bash observal reconcile --dry-run observal auth status observal ops telemetry status ``` Session records remain in the local outbox until the server acknowledges them. *** ## Harness Recommendations ### VS Code Install these extensions for the best experience: * **Python** (`ms-python.python`), LSP, type checking * **Pylance** (`ms-python.vscode-pylance`), fast type inference * **Ruff** (`charliermarsh.ruff`), linting and formatting on save * **ESLint** (`dbaeumer.vscode-eslint`), JavaScript/TypeScript linting * **Prettier** (`esbenp.prettier-vscode`), TypeScript formatting * **Docker** (`ms-azuretools.vscode-docker`), container management Recommended `settings.json` additions: ```json { "[python]": { "editor.defaultFormatter": "charliermarsh.ruff", "editor.formatOnSave": true }, "[typescript]": { "editor.defaultFormatter": "esbenp.prettier-vscode", "editor.formatOnSave": true }, "python.analysis.typeCheckingMode": "basic" } ``` ### PyCharm / IntelliJ 1. Open the project root as a PyCharm project. 2. Set the Python interpreter to the uv-managed venv: `observal-server/.venv/bin/python`. 3. Mark `observal-server` as the sources root. 4. Install the **Ruff** plugin for in-editor linting. 5. Enable **File Watchers** to run `ruff format` on save. *** ## Pre-commit Hooks Install once with `make hooks`. The hooks run automatically on every `git commit`: | Hook | What it does | | ------------------------------------------ | ------------------------------------------------------- | | `ruff` | Lints Python, auto-fixes what it can | | `ruff-format` | Formats Python in place | | `trailing-whitespace` | Strips trailing whitespace | | `end-of-file-fixer` | Ensures files end with a newline | | `check-yaml` / `check-toml` / `check-json` | Validates config file syntax | | `check-added-large-files` | Blocks files over 500KB | | `check-merge-conflict` | Blocks leftover `<<<<<<<` markers | | `detect-private-key` | Scans for private key material | | `no-commit-to-branch` | Blocks direct commits to `main` | | `check-secrets` | Scans staged content for API keys, tokens, `.env` files | | `check-migrations` | Validates Alembic migration chain integrity | | `spdx-update` | Adds your SPDX copyright line to staged files | | `hadolint-docker` | Lints Dockerfiles | If a hook fails, fix the reported issue and commit again. To bypass in an emergency (not recommended): ```bash git commit --no-verify -m "your message" ``` *** ## Pull Request Checklist Before opening a PR: * [ ] `make test` passes locally * [ ] `make lint` passes with no errors * [ ] Branch is rebased on the latest `main` * [ ] Commits follow [Conventional Commits](https://www.conventionalcommits.org/) format * [ ] All commits are signed off with SPDX headers (pre-commit hook handles this automatically) * [ ] PR template is filled in completely and reviewed by the accountable human contributor * [ ] CHANGELOG.md updated under `[Unreleased]` for user-facing changes * [ ] Frontend changes include screenshots in the PR body * [ ] AI-assisted contributions labelled with tool name and version (see [AI Policy](/contributing/ai_policy)) * [ ] No unattended agent submissions without meaningful human direction and review (see [AI Policy](/contributing/ai_policy)) *** ## Getting Help If you are stuck, the best place to ask is **#contributing** on [Discord](https://discord.observal.io). Search the channel history first, most common questions have been answered there already. For bugs use **#bug**. For feature ideas use **#feature-requests**. > \[!NOTE] Do not open a GitHub issue just to ask a question. Issues are for confirmed bugs and accepted feature requests. Questions belong on Discord. # Testing Guide This guide defines the pattern for new Python tests in Observal. The current test suite has several styles because it grew over time. Do not rewrite old tests only for style. New tests and touched files should move toward this pattern when it keeps the diff focused. ## Goals Good tests in this repo should be: * **Hermetic**: no real network, Docker, user config, or external services. * **Behavioral**: assert user-visible behavior or service contracts, not incidental internals. * **Small**: one route group, command, service, or behavior area per file. * **Explicit**: setup is visible in the test or in small local helpers. * **Fast**: sleeps, external calls, and expensive setup are mocked at the boundary. ## File layout Use this structure for most new test files: ```python # SPDX-FileCopyrightText: 2026 Your Name # SPDX-License-Identifier: Apache-2.0 """Tests for the behavior under test. Cover the most important scenarios in a short list when useful: * successful path * denied access * edge case or regression """ from __future__ import annotations import uuid from unittest.mock import MagicMock def _make_item(*, name: str = "example") -> MagicMock: item = MagicMock() item.id = uuid.uuid4() item.name = name return item class TestExampleBehavior: """Tests for the public behavior of ExampleBehavior.""" def test_returns_name_for_valid_item(self) -> None: from services.example import item_name item = _make_item(name="alpha") result = item_name(item) assert result == "alpha" ``` Keep the order consistent: 1. SPDX header 2. Module docstring 3. `from __future__ import annotations` when useful 4. Standard library imports 5. Test library imports 6. App imports needed for the test harness 7. Local helper factories 8. Test classes or test functions ## File scope Prefer one file per behavior area: * `tests/test_agent_name_lookup.py` for agent lookup behavior * `tests/test_component_versions_api.py` for component version endpoints * `observal_cli/tests/test_cmd_scan.py` for the `scan` command * `observal-server/tests/test_jwt.py` for JWT service behavior Split a file when it mixes unrelated layers. For example, schema validation, route behavior, config generation, and resolver service behavior should usually live in separate files. ## Imports Use top-level imports for the test harness: * Standard library modules * `pytest` * `unittest.mock` * Test clients such as `AsyncClient`, `ASGITransport`, and `CliRunner` * Models, schemas, routers, and dependency functions needed by local helpers Inline imports are useful for the unit under test: ```python def test_resolves_agent_components() -> None: from services.agent_resolver import ResolvedAgent resolved = ResolvedAgent( agent_id=uuid.uuid4(), agent_name="demo", agent_version="1.0.0", ) assert resolved.ok is True ``` Use inline imports when importing the unit under test needs environment setup, has side effects, or makes the test easier to isolate. Do not force every import to be inline. ## Helpers and fixtures Prefer small helper functions for local setup: ```python def _make_user(*, role: UserRole = UserRole.user) -> User: user = MagicMock(spec=User) user.id = uuid.uuid4() user.role = role user.email = "test@example.com" user.username = "testuser" return user def _mock_db() -> AsyncMock: db = AsyncMock() db.add = MagicMock() db.commit = AsyncMock() db.refresh = AsyncMock() db.flush = AsyncMock() return db ``` Use `pytest.fixture` when pytest lifecycle behavior is helpful: * Temporary directories through `tmp_path` * Environment isolation through `monkeypatch` * Cleanup with `yield` * Shared setup used by many files through `conftest.py` Prefer helpers over fixtures when the setup takes parameters or is only used in one file. ## Test body shape Use a simple arrange, act, assert flow with blank lines between phases: ```python def test_owner_can_view_pending_listing() -> None: user = _make_user(role=UserRole.user) listing = _make_listing(status=ListingStatus.pending, created_by=user.id) result = can_view_listing(listing, user) assert result is True ``` Comments are only needed when the reason is not obvious. A clear test name is better than a comment that repeats the code. ## Naming Use behavior names: ```python def test_missing_auth_returns_401() -> None: def test_owner_can_install_pending_listing() -> None: def test_scan_does_not_modify_ide_files() -> None: def test_hyphenated_name_resolves_by_name() -> None: ``` Avoid vague names: ```python def test_case_1() -> None: def test_function_works() -> None: def test_route() -> None: ``` A good pattern is: ``` test__ ``` Inside a class, the class can provide the subject: ```python class TestAgentLookup: def test_hyphenated_name_returns_agent(self) -> None: ... ``` Use a test class when there are at least a few related tests. Module-level test functions are fine for one or two simple cases. ## Mocking Mock boundaries, not the behavior under test. Good boundaries to mock: * HTTP clients and webhooks * Database session methods * Filesystem writes when the write itself is not under test * Sleep and time delays * Auth providers * External CLIs and subprocesses Avoid mocking: * Pydantic schemas * Small pure functions * The function being tested * Several layers of internal implementation at once Use `AsyncMock` for async methods and `MagicMock` for sync methods. Use `spec` when mocking model objects: ```python user = MagicMock(spec=User) ``` Keep patches close to the test that needs them: ```python @patch("api.routes.agent.crud._load_agent") async def test_name_lookup_returns_agent(mock_load: MagicMock) -> None: ... ``` ## Async tests Mark async tests explicitly: ```python @pytest.mark.asyncio async def test_fetches_current_user() -> None: from services.users import fetch_current_user db = _mock_db() user = await fetch_current_user(db, user_id=uuid.uuid4()) assert user is not None ``` The root pytest config sets asyncio mode to auto, but the marker keeps async tests easy to spot. ## API route tests Use a small FastAPI app with dependency overrides. Do not boot the full application unless the integration boundary is the thing being tested. ```python import uuid from unittest.mock import AsyncMock, MagicMock import pytest from fastapi import FastAPI from httpx import ASGITransport, AsyncClient from api.deps import get_current_user, get_db from api.routes.agent import router from models.user import User, UserRole def _make_user() -> User: user = MagicMock(spec=User) user.id = uuid.uuid4() user.role = UserRole.user user.email = "test@example.com" user.username = "testuser" return user def _mock_db() -> AsyncMock: db = AsyncMock() db.execute = AsyncMock() db.commit = AsyncMock() return db def _app_with( *, user: User | None = None, db: AsyncMock | None = None, ) -> tuple[FastAPI, AsyncMock, User]: user = user or _make_user() db = db or _mock_db() app = FastAPI() app.include_router(router) app.dependency_overrides[get_current_user] = lambda: user app.dependency_overrides[get_db] = lambda: db return app, db, user class TestAgentLookup: """Tests for agent lookup routes.""" @pytest.mark.asyncio async def test_unknown_agent_returns_404(self) -> None: app, db, _ = _app_with() db.execute.return_value = MagicMock(one_or_none=MagicMock(return_value=None)) async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: response = await client.get("/api/v1/agents/missing-agent") assert response.status_code == 404 ``` For route tests, assert the response status and response body first. Assert database calls only when they are part of the contract. ## CLI tests CLI tests should not read or write the real home directory. Redirect home and current working directory to a temp path: ```python from __future__ import annotations import json from typing import TYPE_CHECKING import pytest from typer.testing import CliRunner from observal_cli.main import app if TYPE_CHECKING: from pathlib import Path runner = CliRunner() @pytest.fixture() def sandbox_home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: monkeypatch.setenv("HOME", str(tmp_path)) monkeypatch.setenv("USERPROFILE", str(tmp_path)) monkeypatch.chdir(tmp_path) return tmp_path def _write_json(path: Path, data: dict) -> None: path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(data, indent=2)) class TestScanCommand: """Tests for the scan command.""" def test_empty_home_exits_with_message(self, sandbox_home: Path) -> None: result = runner.invoke(app, ["scan"]) assert result.exit_code == 1, result.output assert "No harness configurations found" in result.output ``` Always include `result.output` in exit code assertions so failures are easy to diagnose. ## Parametrize Use parametrization for the same behavior across multiple inputs: ```python @pytest.mark.parametrize( ("role", "expected"), [ (UserRole.admin, True), (UserRole.reviewer, True), (UserRole.user, False), ], ) def test_role_can_review(role: UserRole, expected: bool) -> None: result = can_review(role) assert result is expected ``` Do not use a large parameter matrix when separate named tests would be easier to read. ## Property-based tests `hypothesis` is available for invariants that should hold over many inputs. Use it for pure logic, parsers, serializers, redaction, and validation edge cases. ```python from hypothesis import given, strategies as st class TestRedactionProperties: """Property tests for support bundle redaction.""" @given(secret=st.text(min_size=1, max_size=100)) def test_secret_value_is_removed(self, secret: str) -> None: from observal_cli.support.redaction import redact_text result = redact_text(f"token={secret}", secrets=[secret]) assert secret not in result ``` Keep property-based tests deterministic and focused. If a failing example reveals a bug, add a named regression test too. ## Fuzzing Property tests cover invariants you can name. Fuzzing covers the ones you cannot: the Atheris targets live in `fuzz/`, and the project is being submitted to [Google OSS-Fuzz](https://google.github.io/oss-fuzz/) for continuous fuzzing. Reach for a fuzz target instead of a property test when the boundary takes untrusted bytes and the interesting failures are crashes rather than wrong answers. Session transcripts, the ingest classifiers, and both redaction layers are covered today. ```bash make test-fuzz # smoke-test every target python3 fuzz/session_jsonl_fuzzer.py -atheris_runs=100000 python3 fuzz/session_jsonl_fuzzer.py crash- # reproduce a finding ``` `fuzz/README.md` covers adding a target, seed corpora, dictionaries, and the OSS-Fuzz project configuration. ## Assertions Use plain `assert`. Assert the public result before internal interactions: ```python assert response.status_code == 200 assert response.json()["name"] == "my-agent" ``` Use call assertions when the call is the behavior: ```python mock_post.assert_called_once_with("/ingest", json=payload) ``` For CLI tests, include output in the assertion message: ```python assert result.exit_code == 0, result.output ``` For exceptions, assert the important fields: ```python with pytest.raises(HTTPException) as exc_info: await require_admin(user) assert exc_info.value.status_code == 403 assert "admin" in exc_info.value.detail.lower() ``` ## Test directories Use the existing directories: * `tests/` for cross-cutting backend, CLI, and integration-style unit tests * `observal-server/tests/` for server-focused tests that live with the server package * `observal_cli/tests/` for CLI command and CLI package tests * `tests/e2e/` for Playwright tests that require the running stack Shared setup should stay small: * `tests/conftest.py` adds the server source path for imports. * `observal-server/tests/conftest.py` sets test environment variables and shared auth fixtures. Extend `conftest.py` only when setup is useful across several files. ## Running tests Use the make targets for normal workflows: ```bash make test make test-v make test-eval-completeness make test-adversarial make test-fuzz make test-all ``` Run focused pytest commands from `observal-server` when iterating on one file: ```bash cd observal-server uv run pytest ../tests/test_agent_name_lookup.py -q uv run pytest ../observal-server/tests/test_jwt.py -q uv run pytest ../observal_cli/tests/test_cmd_scan.py -q ``` Use `make lint` and `make format` before pushing Python changes. Use `make check` when you need the full pre-commit suite. ## Formatting and lint notes The root `pyproject.toml` relaxes a few lint rules for tests. Those ignores are for practical mock setup, not a reason to leave tests messy. Current per-file ignores include: ```toml [tool.ruff.lint.per-file-ignores] "tests/*" = ["F841", "RUF059", "B007", "TID251"] "observal-server/tests/*" = ["F841", "RUF059", "B007", "TID251", "E402"] ``` In practice: * Unused mock variables are allowed in test files. * Some server tests can use guarded imports after setup. * Tests should import feature modules from their normal server package paths. * New tests should still prefer clear imports, typed helpers, and small setup. ## Review checklist Before submitting a test change, check that: * The test fails without the product fix or covers a meaningful invariant. * External services are mocked or replaced with local fakes. * Real user config and real home directories are not touched. * The test name states the behavior. * Helpers are local unless they are reused across files. * Async tests use `AsyncMock` for awaited methods. * CLI tests include command output in exit code assertions. * API route tests override dependencies in a local FastAPI app. * Focused tests pass locally. # Fuzzing Guide Observal is fuzzed with [Atheris](https://github.com/google/atheris) and is being submitted to [Google OSS-Fuzz](https://google.github.io/oss-fuzz/) for continuous fuzzing; the upstream project is not live yet. The fuzz targets live here so they are versioned alongside the code they exercise; OSS-Fuzz only stores the three-file project configuration, which is mirrored under `fuzz/oss-fuzz/`. ## Targets | Target | Trust boundary | | -------------------------- | ----------------------------------------------------------------------------------------------------- | | `session_jsonl_fuzzer` | Harness session transcripts: `ingest_classify` on the write path, `parse_raw_events` on the read path | | `session_structure_fuzzer` | Same pipeline, driven by Hypothesis-generated transcript records instead of raw bytes | | `secrets_redactor_fuzzer` | `services.secrets_redactor.redact_secrets`, applied to every line and preview before storage | | `support_redaction_fuzzer` | `observal_cli.support.redaction`, the single chokepoint for `observal doctor support bundle` | Every target runs entirely in-process. None of them opens a socket, reads credentials, touches PostgreSQL, ClickHouse or Redis, or depends on the clock, so a crash reproduces from its input alone. `session_jsonl_fuzzer` reads the first input byte as a harness selector (`byte % len(HARNESSES)`) and treats the rest as the transcript, which is why each seed corpus file starts with a single digit and is otherwise plain JSONL. ## Running a target locally ```bash pip install atheris hypothesis python3 fuzz/session_jsonl_fuzzer.py -atheris_runs=100000 ``` Point libFuzzer at the seed corpus and dictionary to start from useful inputs and keep anything new it discovers: ```bash python3 fuzz/session_jsonl_fuzzer.py \ -dict=fuzz/dictionaries/session_jsonl_fuzzer.dict \ /tmp/observal-corpus fuzz/corpus/session_jsonl_fuzzer ``` `session_structure_fuzzer` is a polyglot. Run it under pytest to replay and shrink any example Hypothesis has already recorded: ```bash pytest fuzz/session_structure_fuzzer.py ``` `make test-fuzz` runs a short campaign over every target and is the quickest way to confirm the harnesses still build after a parser change. ## Reproducing a crash A crashing input is a plain file. OSS-Fuzz attaches one to every report; locally libFuzzer writes `crash-` into the working directory. ```bash python3 fuzz/session_jsonl_fuzzer.py crash- ``` Atheris prints the Python traceback and exits non-zero. For an OSS-Fuzz report, `infra/helper.py reproduce observal ` runs the same input inside the container the report came from. ## Adding a target 1. Add `fuzz/_fuzzer.py`. `build.sh` globs `*_fuzzer.py`, so no build change is needed. 2. Import the code under test inside `with atheris.instrument_imports():` so Atheris can add coverage instrumentation as the modules load. Call `_paths.add_source_roots()` first if the target touches `observal-server`. 3. Define `TestOneInput(data: bytes)`, wire it up in `main()`, and keep the harness thin -- decode the input, call the boundary, assert the contract. Expected rejections (a decode error on malformed input) should return; every other exception is a finding. 4. Bound the input size. Slow inputs are reported as timeouts rather than as bugs. 5. Declare any third-party import in the `fuzz` dependency group in `pyproject.toml` and commit the refreshed `uv.lock`. OSS-Fuzz installs that group into the base image's system interpreter, which is the only environment PyInstaller resolves imports against; a package that is merely importable in a local virtualenv is silently omitted from the built target. 6. Add a seed corpus under `corpus/_fuzzer/` and a dictionary at `dictionaries/_fuzzer.dict`. Both are optional and both are picked up automatically. 7. Corpus files are raw fuzzer input, not source. Nothing may rewrite them. `scripts/update_spdx_copyright.py` skips `fuzz/corpus/**`, and `REUSE.toml` carries their licensing instead. `session_jsonl_fuzzer` reads its first byte, so a stray header would repoint the whole corpus at one parser; `tests/test_fuzz_targets.py` asserts every parser still has a seed. 8. Keep seed corpora free of anything that looks like a credential. The repository runs Gitleaks and a pre-commit secret scan; put distinctive vendor prefixes in the dictionary, where they are too short to match a scanner rule, and use zero-entropy placeholders in corpus files. Prefer extending an existing target over adding a near-duplicate. A new target should reach a boundary the current four do not. ## Maintaining the OSS-Fuzz project `fuzz/oss-fuzz/` mirrors `projects/observal/` in [google/oss-fuzz](https://github.com/google/oss-fuzz). Edit the files under `fuzz/oss-fuzz/`, then copy them upstream in the same change so the two never drift. Validate a change against the real builder before opening the upstream pull request: ```bash git clone --depth 1 https://github.com/google/oss-fuzz.git cp -r fuzz/oss-fuzz oss-fuzz/projects/observal cd oss-fuzz python3 infra/helper.py build_image observal python3 infra/helper.py build_fuzzers observal /path/to/Observal # builds this checkout python3 infra/helper.py check_build observal python3 infra/helper.py run_fuzzer observal session_jsonl_fuzzer -- -runs=10000 python3 infra/helper.py build_fuzzers --sanitizer coverage observal /path/to/Observal python3 infra/helper.py coverage observal --fuzz-target=session_jsonl_fuzzer ``` Passing the checkout path to `build_fuzzers` replaces the `git clone` in the Dockerfile with the local tree, which is how you test targets before they land on `main`. Two side effects to know about: the build writes `*.pkg.spec` into the checkout, and the coverage build prepends a `coverage` stub to each target source in place. Only `*.pkg.spec` and `coverage_wrapper.py` are gitignored. Target sources are tracked, so inspect `git diff fuzz/` and restore any modified targets before committing. Once the first hosted build succeeds, add the status badge to the README: ```markdown [![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/observal.svg)](https://issues.oss-fuzz.com/issues?q=project:observal) ``` # AI Policy Observal welcomes the use of AI coding tools. They can meaningfully accelerate development and help contributors tackle complex changes. At the same time, we have zero tolerance for **slop**: unreviewed, low-effort output that wastes reviewer time and degrades the codebase. This policy tells you what is expected when AI tools are part of your workflow. > \[!NOTE] This policy was informed by the [AnkiDroid AI Policy](https://github.com/ankidroid/Anki-Android/blob/main/AI_POLICY.md), adapted with a more permissive stance that reflects the nature of an AI-native project. Attribution is given with thanks. *** ## Fully autonomous contributions without human authorship are not permitted **Unattended agents that independently make material implementation choices and submit code without meaningful human direction, review, and accountability are not allowed to contribute to this project.** Interactive coding agents and assistants, including Claude Code, Pi, Cursor, Copilot, and similar tools, are allowed when an accountable human directs the work, reviews the complete change, can explain it, and explicitly authorizes publication. The distinction is meaningful human authorship and accountability, not whether the tool can edit files, run commands, commit, push, or open a pull request. This is not a quality judgement, it is a legal one. The [US Copyright Office's January 2025 report on AI copyrightability](https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-2-Copyrightability-Report.pdf) explains that material without sufficient human-authored expressive elements may not qualify for copyright protection. Copyrightability is fact-specific, and prompting alone does not necessarily establish human authorship. For this project, unattended submissions create three structural risks: 1. **The CLA may be unsupported.** Our CLA requires the contributor to assert that the contribution is their original creation and that they have the right to license it. An unattended submission provides no accountable human who can make and support that assertion. 2. **The license chain becomes uncertain.** Apache-2.0 depends on contributors having rights they can grant. Insufficient human authorship can leave the project unable to establish a reliable licensing chain for the contribution. 3. **Training data provenance is unknowable.** Autonomous agents may reproduce verbatim or substantially similar GPL-licensed code from their training data without attribution, a risk confirmed by active litigation (*Doe 1 v. GitHub, Inc.*, N.D. Cal. 2022) and studied by the [Software Freedom Conservancy](https://sfconservancy.org/blog/2022/feb/03/github-copilot-copyleft-gpl/) and [FSF](https://www.fsf.org/news/publication-of-the-fsf-funded-white-papers-on-questions-around-copilot). This is the same position taken by [curl](https://curl.se/dev/contribute.html) and documented by the FSF and SFC. **Any PR produced and submitted by an unattended agent without an accountable human author will be closed immediately.** > \[!NOTE] Using an interactive AI coding tool under human direction is explicitly welcome. A human may ask the tool to edit code, run checks, prepare commits, push a branch, draft pull request content, or publish an approved pull request. The human must review the result, make or approve the material choices, accept responsibility for the contribution, and satisfy the CLA. *** ## What is allowed * Using AI tools (Copilot, Cursor, Claude Code, Pi, etc.) to write, refactor, review, or test code, provided you direct the work and review and own the result * Using AI tools to help understand the codebase, generate test cases, draft documentation, or prepare pull request content * Authorizing an interactive coding agent to commit, push, create or update a pull request, or perform other repository actions after you have reviewed the relevant changes and content * Submitting AI-assisted contributions, provided all requirements below are met *** ## Requirements for AI-assisted contributions ### You must be able to explain every line If a reviewer asks you to explain a change, you must be able to do so clearly and accurately. "The AI wrote it" is not an acceptable answer. If you cannot explain it, do not submit it. ### It must compile and the tests must pass Run `make test` and verify CI passes before opening a PR. Do not submit code you have not executed locally. ### Read through the full diff yourself Before opening a PR, read every line of your diff. AI tools make confident-looking mistakes. You are responsible for catching them. ### Frontend changes require screenshots If your PR touches the web frontend, attach screenshots of all affected screens to the PR body. This is required regardless of whether the change was AI-assisted. ### Label AI use and include the tool version If AI tools made a nontrivial contribution to your PR, state so in the PR description. Include the tool name and version (for example: `Claude Sonnet 4`, `GPT-4o`, `Cursor 0.48`). A nontrivial contribution means the AI wrote, restructured, or significantly modified the code, not just autocomplete suggestions. *** ## What is not allowed * Unattended coding agents choosing work, implementing it, and submitting PRs without meaningful human direction and review * Publishing AI-generated GitHub comments, review replies, or PR template content without human review and explicit approval * Submitting output you have not read and understood * Claiming human authorship when no accountable human made or approved the material creative choices * Repeating the same AI-generated mistakes across multiple PRs after being told about them > \[!WARNING] PRs that show clear signs of unreviewed AI output, boilerplate that does not match the codebase, incorrect variable names, placeholder text, hallucinated API calls, will be **closed without review**. A second instance may result in a contribution ban.