# Development Guide Source: https://docs.observal.io/DEVELOPMENT_GUIDE > [!IMPORTANT] > > **Discord is the primary place to ask questions.** Join at [discord.observal.io](https://discord.observal.io). > > * **#contributing**, setup help, workflow questions, anything about the contribution process > * **#bug**, discuss bugs before filing a GitHub issue > * **#feature-requests**, pitch ideas and get early feedback before writing code > > GitHub issues and PRs are for concrete, actionable items. Exploratory discussion belongs on Discord first. > Parts of this guide structure were inspired by the [AnkiDroid Development Guide](https://github.com/ankidroid/Anki-Android/wiki/Development-Guide). Attribution given with thanks, check them out if you want another welcoming OSS project to contribute to. *** ## Table of Contents * [Community Standards](/DEVELOPMENT_GUIDE#community-standards) * [Prerequisites](/DEVELOPMENT_GUIDE#prerequisites) * [Git Setup](/DEVELOPMENT_GUIDE#git-setup) * [SSH authentication](/DEVELOPMENT_GUIDE#ssh-authentication) * [Git identity](/DEVELOPMENT_GUIDE#git-identity) * [Fork and clone](/DEVELOPMENT_GUIDE#fork-and-clone) * [First-time Setup](/DEVELOPMENT_GUIDE#first-time-setup) * [Install pre-commit hooks](/DEVELOPMENT_GUIDE#install-pre-commit-hooks) * [Start the Docker stack](/DEVELOPMENT_GUIDE#start-the-docker-stack) * [Install the CLI](/DEVELOPMENT_GUIDE#install-the-cli) * [Demo accounts](/DEVELOPMENT_GUIDE#demo-accounts) * [Make Targets Reference](/DEVELOPMENT_GUIDE#make-targets-reference) * [Architecture Overview](/DEVELOPMENT_GUIDE#architecture-overview) * [Git Workflow](/DEVELOPMENT_GUIDE#git-workflow) * [Making a new branch](/DEVELOPMENT_GUIDE#making-a-new-branch) * [Keeping your branch up to date](/DEVELOPMENT_GUIDE#keeping-your-branch-up-to-date) * [Dealing with merge conflicts](/DEVELOPMENT_GUIDE#dealing-with-merge-conflicts) * [Submitting a pull request](/DEVELOPMENT_GUIDE#submitting-a-pull-request) * [After review feedback](/DEVELOPMENT_GUIDE#after-review-feedback) * [Working on the Backend](/DEVELOPMENT_GUIDE#working-on-the-backend) * [Running tests](/DEVELOPMENT_GUIDE#running-tests) * [Testing conventions](/DEVELOPMENT_GUIDE#testing-conventions) * [Running a single test](/DEVELOPMENT_GUIDE#running-a-single-test) * [Code coverage](/DEVELOPMENT_GUIDE#code-coverage) * [Adding a database migration](/DEVELOPMENT_GUIDE#adding-a-database-migration) * [Connecting to PostgreSQL directly](/DEVELOPMENT_GUIDE#connecting-to-postgresql-directly) * [Connecting to ClickHouse directly](/DEVELOPMENT_GUIDE#connecting-to-clickhouse-directly) * [Debugging the API](/DEVELOPMENT_GUIDE#debugging-the-api) * [Working on the Frontend](/DEVELOPMENT_GUIDE#working-on-the-frontend) * [Running the frontend in isolation](/DEVELOPMENT_GUIDE#running-the-frontend-in-isolation) * [Design system](/DEVELOPMENT_GUIDE#design-system) * [Adding a new API endpoint to the frontend](/DEVELOPMENT_GUIDE#adding-a-new-api-endpoint-to-the-frontend) * [Screenshots for UI changes](/DEVELOPMENT_GUIDE#screenshots-for-ui-changes) * [Working on the CLI](/DEVELOPMENT_GUIDE#working-on-the-cli) * [Reinstalling after changes](/DEVELOPMENT_GUIDE#reinstalling-after-changes) * [Harness Recommendations](/DEVELOPMENT_GUIDE#harness-recommendations) * [VS Code](/DEVELOPMENT_GUIDE#vs-code) * [PyCharm / IntelliJ](/DEVELOPMENT_GUIDE#pycharm--intellij) * [Pre-commit Hooks](/DEVELOPMENT_GUIDE#pre-commit-hooks) * [Pull Request Checklist](/DEVELOPMENT_GUIDE#pull-request-checklist) * [Getting Help](/DEVELOPMENT_GUIDE#getting-help) *** ## Community Standards We are a small, active community and we take the quality of interactions seriously. > [!WARNING] > > The following will result in a **moderator warning**. A second violation results in a **temporary or permanent ban** from the repository and Discord: > > * Pinging contributors, maintainers, or reviewers unnecessarily (outside of a direct reply on your own open PR or issue) > * Submitting low-effort or unreviewed PRs (slop), including unreviewed AI output or autonomous agent submissions > * Violating the [Code of Conduct](https://github.com/Observal/Observal/blob/main/CODE_OF_CONDUCT.md) in any channel > * Harassing reviewers over merge timelines Maintainers volunteer their time. Treat them accordingly. ### A note on autonomous coding agents **Autonomous coding agents (Devin, SWE-agent, OpenHands, etc.) are not permitted to submit PRs.** This is a legal constraint, not a quality judgment. The US Copyright Office's 2025 guidance confirms that purely AI-generated code has no copyright owner, which breaks our CLA and our ability to license and enforce the project. See the [AI Policy](https://github.com/Observal/Observal/blob/main/AI_POLICY.md) for the full explanation. *** ## Prerequisites | Tool | Version | Install | | - | - | - | | Docker | 24+ with Compose v2 | [docs.docker.com](https://docs.docker.com/get-docker/) | | uv | latest | `curl -LsSf https://astral.sh/uv/install.sh \| sh` | | Node.js | 22.6+ | Via [nvm](https://github.com/nvm-sh/nvm) or [mise](https://mise.jdx.dev/) | | pnpm | 10+ | `npm install -g pnpm` | | Git | 2.28+ | [git-scm.com](https://git-scm.com/) | Verify your Docker installation supports Compose v2: ```bash theme={null} docker compose version # should print: Docker Compose version v2.x.x ``` If you see `docker-compose: command not found` or a v1 version, upgrade Docker Desktop or install the Compose plugin separately. *** ## Git Setup ### SSH authentication Using SSH is strongly recommended, it avoids password prompts and is required if you enable 2FA on GitHub. ```bash theme={null} # Generate a key (skip if you already have one at ~/.ssh/id_ed25519) ssh-keygen -t ed25519 -C "your@email.com" # Copy the public key to your clipboard cat ~/.ssh/id_ed25519.pub ``` Add the output to **GitHub → Settings → SSH and GPG keys → New SSH key**. Test it: ```bash theme={null} ssh -T git@github.com # Hi your-username! You've successfully authenticated... ``` ### Git identity Set your identity so it matches your GitHub account: ```bash theme={null} git config --global user.name "Your Name" git config --global user.email "your@email.com" ``` If you want to sign commits with GPG as well (optional but appreciated): ```bash theme={null} git config --global commit.gpgsign true git config --global user.signingkey YOUR_GPG_KEY_ID ``` ### Fork and clone You only need to do this once. For every subsequent contribution, just create a new branch on your existing fork. 1. Fork the repository on GitHub (top-right **Fork** button). 2. Clone your fork using SSH: ```bash theme={null} git clone git@github.com:YOUR-USERNAME/Observal.git cd Observal ``` 3. Add the upstream remote so you can pull in changes from the main repo: ```bash theme={null} git remote add upstream https://github.com/Observal/Observal.git ``` 4. Verify your remotes: ```bash theme={null} git remote -v # origin git@github.com:YOUR-USERNAME/Observal.git (fetch) # origin git@github.com:YOUR-USERNAME/Observal.git (push) # upstream https://github.com/Observal/Observal.git (fetch) # upstream https://github.com/Observal/Observal.git (push) ``` *** ## First-time Setup ### Install pre-commit hooks Do this **before your first commit**. The hooks run ruff, format checkers, secret scanners, SPDX header injection, and migration chain validation automatically. ```bash theme={null} make hooks ``` ### Start the Docker stack ```bash theme={null} cp .env.example .env make up ``` The first build takes several minutes while Docker downloads and compiles images. Subsequent starts are fast. Watch progress with: ```bash theme={null} make logs ``` For code, dependency, and Compose changes, use: ```bash theme={null} make rebuild ``` `make rebuild-fast` is the same target under another name. It is not a separate dev profile and does not enable hot reload. It builds only the two app images that contain local code: * `observal-api`, shared by the API, init, and worker services * `observal-web`, used by the web service Then it starts the full stack with the freshly built images. Wait until all services are healthy: ```bash theme={null} docker compose -f docker/docker-compose.yml ps # All services should show "healthy" or "running" ``` **Default endpoints:** | Service | URL | | - | - | | LB (all traffic) | `http://localhost` | | Web UI (direct) | `http://localhost:3000` | | Prometheus, optional | `http://localhost:9090` | | Grafana, optional | `http://localhost:3001` | | ClickHouse HTTP | `http://localhost:8123` | ### Install the CLI ```bash theme={null} uv tool install --editable . ``` ### Log in ```bash theme={null} observal auth login ``` On a fresh server, this bootstraps the admin account automatically. Use `super@demo.example` / `super-changeme` for the super\_admin account. ### Demo accounts Seeded automatically on first startup: | Email | Password | Role | | - | - | - | | `super@demo.example` | `super-changeme` | super\_admin | | `admin@demo.example` | `admin-changeme` | admin | | `dev@demo.example` | `dev-changeme` | developer | | `user@demo.example` | `user-changeme` | user | *** ## Make Targets Reference ```bash theme={null} make up # start the full Docker stack make down # stop the stack make rebuild # rebuild app images and restart (alias: make rebuild-fast) make logs # tail logs from all services make test # run the test suite quickly make test-v # verbose test output make lint # ruff check + hadolint make format # ruff format (auto-fix) make check # pre-commit on all files make hooks # install pre-commit hooks make reset # nuke all volumes and rebuild from scratch (destructive) ``` ### Which rebuild target should I use? | Situation | Target | Notes | | - | - | - | | Any code, dependency, schema, init, worker, or Compose change | `make rebuild` | `make rebuild-fast` is the same target. Rebuilds the shared API image (API, init, worker) and the web image, then recreates changed services. | | Cache looks stale or the stack behaves unexpectedly | `make rebuild-clean` | Rebuilds from scratch without the Docker cache and removes volumes. | `make down` stops containers started by `make rebuild`. *** ## Architecture Overview Observal is a monorepo: ``` observal-server/ FastAPI backend (Python) observal_cli/ CLI and session exporters (Python) web/ Vite 8 / React 19 frontend (TypeScript) tests/ Shared test suite (~1500 tests, 96 files) docker/ Docker Compose and Dockerfiles docs/ Documentation scripts/ Dev tooling scripts ``` **Databases:** * **PostgreSQL**, relational data (users, agents, registry, feedback) * **ClickHouse**, session events, aggregates, audit events, and security events They are not interchangeable. Never write telemetry to Postgres or relational data to ClickHouse. **Supporting services:** Redis (pub/sub + arq job queue), arq worker, nginx reverse proxy. Prometheus and Grafana are optional. See [AGENTS.md](https://github.com/Observal/Observal/blob/main/AGENTS.md) for a complete map of every important file and service. *** ## Git Workflow ### Making a new branch Always branch from the latest `main`. Never commit directly to `main`. ```bash theme={null} git checkout main git pull upstream main git checkout -b feature/my-feature # or: fix/my-bug, docs/my-doc ``` ### Keeping your branch up to date If `main` has moved forward while you were working, rebase your branch onto it: ```bash theme={null} git checkout main git pull upstream main git checkout feature/my-feature git rebase main ``` Rebasing keeps history linear and makes your PR easier to review than a merge commit would. ### Dealing with merge conflicts Conflicts arise when the same file was changed in both `main` and your branch. During `git rebase main`, git will pause and show you: ``` CONFLICT (content): Merge conflict in some/file.py ``` Open the file and look for the three Git conflict-marker lines: seven less-than signs followed by `HEAD`, seven equals signs, and seven greater-than signs followed by the other branch name. The text between them contains the competing changes. Edit the file to the correct final state (keeping whichever changes are right, or combining them), then: ```bash theme={null} git add some/file.py git rebase --continue ``` If you get confused and want to start over: ```bash theme={null} git rebase --abort ``` After resolving and pushing, use `--force-with-lease` rather than `--force`, it's safer: ```bash theme={null} git push origin feature/my-feature --force-with-lease ``` ### Submitting a pull request 1. Make sure your branch is rebased on the latest `main` (see above). 2. Push to your fork: ```bash theme={null} git push origin feature/my-feature ``` 3. GitHub will show a banner on your fork offering to open a PR. Click it, or go to the [Observal repository](https://github.com/Observal/Observal) directly. 4. Fill in the PR template completely. PRs that do not follow the template will be closed. 5. Link the related issue if one exists (`Fixes #123` in the PR body closes it automatically on merge). ### After review feedback If a reviewer requests changes, make them on the same branch and push again. Do not open a new PR. If your changes are small fixes to an existing commit, amend rather than adding new commits: ```bash theme={null} git add the-changed-file.py git commit --amend --no-edit git push origin feature/my-feature --force-with-lease ``` *** ## Working on the Backend ### Running tests Tests mock all external services. Docker does not need to be running. ```bash theme={null} make test make test-v make test-eval-completeness make test-adversarial make test-all # or directly: cd observal-server uv run --with pytest --with pytest-asyncio --with hypothesis --with pyarrow pytest ../tests/ -q ``` ### Testing conventions New Python tests should follow the [Testing Guide](/testing/Testing_Guide). The current suite has mixed historical patterns, so do not rewrite old tests only for style. When adding or touching tests, prefer the clean pattern documented there: one behavior area per file, small local helper factories, hermetic CLI and API test setup, explicit async mocks, and behavior-focused assertions. ### Running a single test ```bash theme={null} cd observal-server uv run pytest ../tests/test_registry_types.py -q # or a specific test function: uv run pytest ../tests/test_registry_types.py::TestMcpRoutes::test_submit_mcp -v ``` ### Code coverage Coverage is collected automatically on the `3.13` matrix run in CI. To generate it locally: ```bash theme={null} cd observal-server uv run pytest ../tests/ --cov=../observal_cli --cov=. --cov-report=html -q # open htmlcov/index.html in your browser ``` ### Adding a database migration ```bash theme={null} bash scripts/new_migration.sh "describe_your_change" ``` Edit the generated file in `observal-server/alembic/versions/`. Then verify the chain is intact: ```bash theme={null} python3 scripts/check_migrations.py ``` > [!CAUTION] > > Never edit an existing migration file. Always create a new one. A broken migration chain blocks CI and prevents the server from starting. Apply migrations to your local stack: ```bash theme={null} docker compose -f docker/docker-compose.yml run --rm init ``` ### Connecting to PostgreSQL directly ```bash theme={null} docker compose -f docker/docker-compose.yml exec db \ psql -U observal -d observal ``` Useful for inspecting tables, running manual queries, or verifying migration results. ### Connecting to ClickHouse directly ClickHouse exposes an HTTP interface. Use the Play UI in your browser: ``` http://localhost:8123/play ``` Or query from the terminal: ```bash theme={null} curl -s "http://localhost:8123/?query=SELECT+count()+FROM+session_events" \ -u "default:" ``` Or connect with the CLI client: ```bash theme={null} docker compose -f docker/docker-compose.yml exec clickhouse \ clickhouse-client --user default ``` ### Debugging the API The API server runs with `--reload` in development mode. Log output is available via: ```bash theme={null} docker compose -f docker/docker-compose.yml logs -f observal-api ``` To add a breakpoint, use `breakpoint()` in Python code, the debugger will pause in the container's stdout. Or attach a remote debugger by adding `debugpy` to your dev dependencies and exposing a debug port. The OpenAPI docs are available at (API port, not through the LB which blocks these paths): * `http://localhost:8000/docs` (Swagger UI, requires direct API port access) * `http://localhost:8000/redoc` (ReDoc, requires direct API port access) > Note: The nginx LB blocks `/docs`, `/redoc`, and `/openapi.json` in production. For local dev, expose the API port directly or use `docker compose exec`. *** ## Working on the Frontend ### Running the frontend in isolation ```bash theme={null} cd web pnpm install pnpm dev ``` The Vite dev server proxies all `/api` calls to the backend at `http://localhost:8000`, configured in `web/vite.config.ts`. The Docker stack does not publish port 8000, so run the API locally or point the proxy target at `http://localhost` (the nginx LB). ### Design system | Token | Value | | - | - | | Color space | OKLCH | | Themes | `light`, `dark`, `midnight`, `forest`, `sunset` | | Display font | Archivo | | Body font | Albert Sans | | Code font | JetBrains Mono | | Spacing base | 4pt | | Components | shadcn/ui | | Charts | Recharts | | Data fetching | TanStack Query | | Tables | TanStack Table | All themes are defined in `web/src/app/globals.css` as CSS custom properties. Semantic tokens (`background`, `foreground`, `card`, `border`, `primary`, `secondary`, `accent`, `destructive`, `success`, `warning`, `info`) map to OKLCH values per theme. When adding new UI, use the semantic tokens, never hardcode colors. Check all five themes look correct before submitting. ### Adding a new API endpoint to the frontend 1. Add the TypeScript response type to `web/src/lib/types.ts`. 2. Add the fetch call to `web/src/lib/api.ts`. 3. Add the TanStack Query hook to `web/src/hooks/use-api.ts`. 4. Use the hook in your component. ### Screenshots for UI changes > [!IMPORTANT] > > Any PR that touches the web frontend must include **screenshots of all affected screens** in the PR description. This is required regardless of how small the change is. Attach screenshots directly to the PR body, not as review comments. *** ## Working on the CLI ### Reinstalling after changes The CLI is installed as a uv tool. Changes to source files are reflected immediately because it is installed in editable mode (`--editable`). If you add new entry points or change `pyproject.toml`, reinstall: ```bash theme={null} uv tool install --editable . --reinstall ``` ### Testing session delivery Run a harness session with hooks installed, then reconcile and inspect exporter status: ```bash theme={null} observal reconcile --dry-run observal auth status observal ops telemetry status ``` Session records remain in the local outbox until the server acknowledges them. *** ## Harness Recommendations ### VS Code Install these extensions for the best experience: * **Python** (`ms-python.python`), LSP, type checking * **Pylance** (`ms-python.vscode-pylance`), fast type inference * **Ruff** (`charliermarsh.ruff`), linting and formatting on save * **ESLint** (`dbaeumer.vscode-eslint`), JavaScript/TypeScript linting * **Prettier** (`esbenp.prettier-vscode`), TypeScript formatting * **Docker** (`ms-azuretools.vscode-docker`), container management Recommended `settings.json` additions: ```json theme={null} { "[python]": { "editor.defaultFormatter": "charliermarsh.ruff", "editor.formatOnSave": true }, "[typescript]": { "editor.defaultFormatter": "esbenp.prettier-vscode", "editor.formatOnSave": true }, "python.analysis.typeCheckingMode": "basic" } ``` ### PyCharm / IntelliJ 1. Open the project root as a PyCharm project. 2. Set the Python interpreter to the uv-managed venv: `observal-server/.venv/bin/python`. 3. Mark `observal-server` as the sources root. 4. Install the **Ruff** plugin for in-editor linting. 5. Enable **File Watchers** to run `ruff format` on save. *** ## Pre-commit Hooks Install once with `make hooks`. The hooks run automatically on every `git commit`: | Hook | What it does | | - | - | | `ruff` | Lints Python, auto-fixes what it can | | `ruff-format` | Formats Python in place | | `trailing-whitespace` | Strips trailing whitespace | | `end-of-file-fixer` | Ensures files end with a newline | | `check-yaml` / `check-toml` / `check-json` | Validates config file syntax | | `check-added-large-files` | Blocks files over 500KB | | `check-merge-conflict` | Blocks leftover `<<<<<<<` markers | | `detect-private-key` | Scans for private key material | | `no-commit-to-branch` | Blocks direct commits to `main` | | `check-secrets` | Scans staged content for API keys, tokens, `.env` files | | `check-migrations` | Validates Alembic migration chain integrity | | `spdx-update` | Adds your SPDX copyright line to staged files | | `hadolint-docker` | Lints Dockerfiles | If a hook fails, fix the reported issue and commit again. To bypass in an emergency (not recommended): ```bash theme={null} git commit --no-verify -m "your message" ``` *** ## Pull Request Checklist Before opening a PR: * [ ] `make test` passes locally * [ ] `make lint` passes with no errors * [ ] Branch is rebased on the latest `main` * [ ] Commits follow [Conventional Commits](https://www.conventionalcommits.org/) format * [ ] All commits are signed off with SPDX headers (pre-commit hook handles this automatically) * [ ] PR template is filled in completely and reviewed by the accountable human contributor * [ ] CHANGELOG.md updated under `[Unreleased]` for user-facing changes * [ ] Frontend changes include screenshots in the PR body * [ ] AI-assisted contributions labelled with tool name and version (see [AI Policy](https://github.com/Observal/Observal/blob/main/AI_POLICY.md)) * [ ] No unattended agent submissions without meaningful human direction and review (see [AI Policy](https://github.com/Observal/Observal/blob/main/AI_POLICY.md)) *** ## Getting Help If you are stuck, the best place to ask is **#contributing** on [Discord](https://discord.observal.io). Search the channel history first, most common questions have been answered there already. For bugs use **#bug**. For feature ideas use **#feature-requests**. > [!NOTE] > > Do not open a GitHub issue just to ask a question. Issues are for confirmed bugs and accepted feature requests. Questions belong on Discord. # ADR 0001: Adopt Agentic Resource Discovery (ARD) as the discovery envelope Source: https://docs.observal.io/adr/0001-agentic-resource-discovery **Status:** Accepted **Date:** 2026-09-20 **Deciders:** Observal maintainers ## Context Observal is install-first: a person builds an Agent from registry components, publishes it, and pulls it into a harness before a session starts. That model is right for repeatable, governed workflows and wrong for the moment a developer is mid-task and needs a capability that already exists in the registry. It also means the assistant only knows about resources someone installed ahead of time, so it rebuilds things that exist and tells users a capability is unavailable when it is one search away. The Agentic Resource Discovery specification (ARD) defines a common way to describe a resource and a common search API so any client can ask "what exists for this task?" across registries. It deliberately does **not** define installation, invocation, authentication, review, or telemetry. Observal already has all of those. The fit is: ARD says what exists, Observal says how to use it safely. This record fixes the decisions the discovery work depends on so later phases can be built against a stable target. The full plan lives outside the repo; this ADR is the part that must not drift. ## Decision 1: Pin the specification Observal implements **ARD v0.91 (Proposal)** as published at commit `b76f235a8f461876ad4f1e77abd0eb0eb302b48d` of [`ards-project/ard-spec`](https://github.com/ards-project/ard-spec) (2026-09-12). The JSON Schema, JSON-LD context, CDDL grammar, OpenAPI file and the official conformance CLI from that commit are vendored unmodified under `observal-server/vendor/ard/` (see `VENDORED.md` there for provenance and the update procedure). The spec is a proposal and will change. Observal tracks it by re-vendoring at a new commit, updating this ADR, and re-running the conformance tests, never by hand-editing vendored files or silently accepting whatever the live site says. ## Decision 2: Native models stay authoritative; ARD is a projection The existing `Agent`/`AgentVersion`, `McpListing`/`McpVersion`, `SkillListing`/`SkillVersion`, hook, prompt and sandbox tables remain the source of truth for installation, lifecycle, review and ownership. Discovery adds a read model, `discovery_entries`, populated from approved versions and refreshed on lifecycle events. Nothing that exists today is replaced or repurposed. The `/api/v1/agents` and component contracts are unchanged. ## Decision 3: Identifier scheme Every native resource gets exactly one permanent ARD identifier: ``` urn:air::: ``` * `` is the host of the deployment's public URL (`deployment.public_url` dynamic setting), which satisfies ARD's requirement that the publisher segment be a fully qualified domain name. * `` is one of `agent`, `mcp`, `skill`, `hook`, `prompt`, `sandbox` and occupies ARD's namespace segment. * `` is the native listing's primary key and occupies ARD's name segment. Example: `urn:air:observal.acme.com:skill:5f2c1a9e-8b1d-4d0c-9c4e-1f2a3b4c5d6e` **Why the UUID and not `namespace/slug`.** ARD requires the identifier to be a stable handle that survives the resource moving. Observal's `namespace/slug` changes on rename and on `transfer-owner`. The UUID never changes. The human-readable identity is still carried, as `obs:nativeRef` (`namespace/slug@version`), so clients can pivot to existing commands. **The publisher domain is pinned.** It is derived from `deployment.public_url` the first time a real fully qualified domain is seen and stored in the `discovery.publisher_domain` setting; after that, identifiers do not follow the public URL if the deployment moves hosts. Until a real domain is configured, identifiers use the placeholder `observal.local` and are expected to change once. Changing the pinned setting deliberately renames every identifier and is an administrator action. The artifact URL, by contrast, is a location and always follows the current public URL. Single-label hosts and IP addresses are never accepted as publishers. Externally ingested entries keep the identifier their publisher assigned. Observal never rewrites another publisher's URN. ## Decision 4: Endpoint base and well-known paths ``` GET /.well-known/ard.json manifest of public, approved entries GET /.well-known/ai-catalog.json predecessor path, same content POST /api/v1/ard/search ARD Search GET /api/v1/ard/agents ARD List (deterministic browse) POST /api/v1/ard/explore ARD Explore (facets), later phase GET /api/v1/ard/entries/{identifier} fetch one full entry (Observal-specific) GET /api/v1/artifacts/{kind}/{uuid}/{version} permanent versioned artifact with digest ``` The registry advertises `https:///api/v1/ard` as its operational base through an entry of type `application/ai-registry+json` in the manifest. ARD defines no operation for fetching a complete entry by identifier; clients are expected to fetch the artifact `url`. Observal adds `/entries/{identifier}` for its own CLI and UI. It is not part of the conformance surface. ## Decision 5: `federation` when omitted The spec's default for the `federation` field is `auto`. Observal honours that literally: an omitted field is treated as `auto`, **bounded by the administrator's upstream allowlist**. With the default empty allowlist this is functionally local-only. When an admin allows upstreams, `auto` and `referrals` only ever reach those. Rejected alternative: default to `none`. That would make a conformant client receive different behaviour than the spec promises, and the privacy goal is achieved just as well by an empty allowlist. Search text may contain sensitive information (incident details, unreleased names, internal service names). It never leaves the deployment unless an admin has allowed a specific upstream, and results from upstreams always carry their `source`. ## Decision 6: Visibility Discovery narrows the signed-in user's registry view to their own namespace and teamspaces they currently belong to. This is intentionally stricter than registry browsing/install for public resources in other namespaces: a public resource can be installed by reference without appearing in a user's ARD search. The same scope applies to search, list, inspect, and versioned artifact reads, including imported A2A agents. Membership is checked at query time, so removing a member revokes their ARD access to that team's public and private entries. Submitter status alone does not grant access to an old teamspace. Two filters apply, in order: 1. **Who can see the listing** — ordinary signed-in users see personal listings they own (public or private) and listings in *all* teamspaces where they are current members (public or private). Global reviewers retain their public review scope; admins and super-admins still see everything. Anonymous calls retain the public-catalog behavior described below. 2. **Which lifecycle states are returned** — approved versions to everyone who passes filter 1; pending and rejected versions only to users whose effective component permission is `owner` (the same rule install uses for owner fallback); archived entries excluded unless the query filters on `obs:lifecycle` explicitly. Unauthenticated requests receive only public, approved entries, and only when the registry-wide `deployment.public_registry_enabled` setting is on (default off), the same switch that governs anonymous browsing elsewhere. When it is off, unauthenticated search returns an empty result set and the well-known manifest carries only the registry's own `application/ai-registry+json` entry, both of which are conformant. The well-known manifest is always computed as an unauthenticated request. ## Decision 7: Media types and compatibility inputs Emitted (canonical) types: | Observal resource | ARD `type` | | - | - | | Skill | `application/ai-skill+md` | | MCP server | `application/mcp-server-card+json` (a descriptor, not the live stream) | | Prompt | `application/vnd.observal.prompt+json` | | Sandbox | `application/vnd.observal.sandbox+json` | | Hook | `application/vnd.observal.hook+json` | | Agent | `application/vnd.observal.agent+json` | | External, unknown | preserved exactly as published | An Observal Agent is an installable configuration, not a running service, so it is never emitted as `application/a2a-agent-card+json`. (ADR 0002 adds remote A2A agents as imported entries of that type, and delegation to both kinds.) Accepted and normalised on input, with the original preserved in `raw_entry`: | Input | Normalised to | | - | - | | `urn:ai:…` | `urn:air:…` | | `application/mcp-server`, `application/mcp-server+json` | `application/mcp-server-card+json` | | `application/ai-skill` | `application/ai-skill+md` | | `/.well-known/ai-catalog.json`, `rel="ai-catalog"` | treated as `/.well-known/ard.json`, `rel="ard"` | ## Decision 8: Relevance is the only thing in `score` ARD's `score` (0–100) is semantic relevance and nothing else. Approval, publisher verification, liveness, harness compatibility and adoption are returned as separate, named fields (`obs:approval`, `obs:trust`, `obs:availability`, `obs:supportedHarnesses`, …) and are never blended into `score`. Policy eligibility is applied as a filter before ranking, not as a weight inside it. ## Decision 9: What is deliberately deferred * **Web ingestion.** ARD §5.2 says every registry MUST crawl entry sources. Observal defers this to the external-sources phase. The vendored conformance tool passes without it; until that phase ships, Observal is an ARD *publisher* and *search endpoint*, not a complete ARD *registry*, and says so here rather than claiming otherwise. * **Explore.** Optional in the spec; added with search-quality work. * **Trust manifest verification.** Entries Observal publishes carry a `trustManifest.identity` of `https://` (an HTTPS FQDN URI string, `identityType: "https"`) bound to the URN's publisher segment. Verification of third-party manifests arrives with external ingestion. ## Decision 10: Search baseline needs no external service The first search implementation is PostgreSQL full-text search plus `pg_trgm` (already enabled). It must remain a complete, supported configuration. Dense retrieval is an optional later addition behind a provider interface and an explicit infrastructure change (pgvector); its absence never disables discovery. ## Consequences * Discovery reaches all ten harnesses through the bundled skill and the CLI, which already ship to every harness on login. No new runtime is required for the first milestone. * Per-harness behaviour (how MCP is installed, whether tools refresh mid-session, whether a hook can inject context) is recorded as verified facts in the shared harness registry and tested, so later phases gate on data rather than assumptions. * Anything that permanently changes a developer's setup, or that can write or delete, keeps asking first. Approved skills and prompts, being text, may be loaded automatically. * Conformance is tested in CI against the vendored tool at the pinned commit. A spec change is a deliberate re-vendor, reviewed like any other dependency update. # observal admin Source: https://docs.observal.io/cli/admin Manage server settings, users, security policy, SSO, audit data, and the submission review queue. All 24 workflows support `--output table|json`. JSON mode never prompts or emits progress banners. Destructive JSON commands require `--force` or `--yes`. Core administration requires the `admin` role. Review commands are also available to global reviewers and authorized teamspace owners or reviewers. Some role changes require `super_admin`. ## Commands | Command | Purpose | | - | - | | `settings` | List dynamic server settings | | `set` | Create or update a dynamic setting | | `users` | List users | | `create-user` | Create a password-auth user | | `reset-password` | Reset or generate a user password | | `delete-user` | Permanently delete a user | | `set-role` | Change a user role | | `diagnostics` | Show database, key, and runtime health | | `trace-privacy` | Show trace-redaction policy | | `trace-privacy-set` | Change trace-redaction policy | | `cache-clear` | Clear server caches | | `saml-config` | Show redacted SAML configuration | | `saml-config-set` | Create or replace SAML configuration | | `saml-config-delete` | Delete SAML configuration | | `scim-tokens` | List SCIM token metadata | | `scim-token-create` | Create a one-time SCIM bearer token | | `scim-token-revoke` | Revoke a SCIM token | | `security-events` | Query security events | | `audit-log` | Query compliance audit events | | `audit-log-export` | Export audit events as CSV or JSON | | `review list` | List pending submissions | | `review show` | Show one submission | | `review approve` | Approve a component, Agent, or bundle | | `review reject` | Reject a component, Agent, or bundle | ## Settings ```bash theme={null} observal admin settings --output json observal admin set review.require_approval true --output json ``` `settings` returns the direct settings array. Sensitive values are redacted by the server. `set` returns the server setting object and never echoes the supplied value in human output. A positional setting value may be retained by shell history. Prefer deployment secret files for sensitive settings that support external management. ## Users and roles List users: ```bash theme={null} observal admin users --output json ``` Create a user and let the server generate a password: ```bash theme={null} observal admin create-user alice@example.com 'Alice Smith' --role user --output json ``` Supported roles are `super_admin`, `admin`, `reviewer`, and `user`. Create with a chosen password only when shell-history exposure is acceptable: ```bash theme={null} observal admin create-user alice@example.com 'Alice Smith' --password 'chosen-password' --output json ``` Reset interactively in human mode: ```bash theme={null} observal admin reset-password alice@example.com ``` Generate a password without prompting: ```bash theme={null} observal admin reset-password alice@example.com --generate --output json ``` JSON reset requires `--generate`. Created and generated passwords are returned once. Treat the entire result as a secret and do not paste it into logs, issues, or chat. Change a role: ```bash theme={null} observal admin set-role alice@example.com reviewer --output json ``` Delete a user: ```bash theme={null} observal admin delete-user alice@example.com --force --output json ``` Human deletion prompts unless `--force` or `--yes` is present. JSON deletion requires confirmation through one of those flags. ## Diagnostics and policy ```bash theme={null} observal admin diagnostics --output json observal admin trace-privacy --output json observal admin trace-privacy-set true --output json observal admin cache-clear --output json ``` `diagnostics` returns the direct health object. Trace privacy responses return `trace_privacy`. Cache clear returns the number of cleared entries. ## SAML Show redacted configuration: ```bash theme={null} observal admin saml-config --output json ``` Create or replace configuration: ```bash theme={null} observal admin saml-config-set \ --idp-entity-id 'https://idp.example.com/entity' \ --idp-sso-url 'https://idp.example.com/sso' \ --idp-x509-cert "$(cat idp-cert.pem)" \ --active \ --output json ``` Every update requires the IdP entity ID, SSO URL, and X.509 certificate. Optional flags include `--idp-slo-url`, `--sp-entity-id`, `--jit|--no-jit`, and `--active|--inactive`. Certificate and private-key material is never returned by configuration reads. Delete configuration: ```bash theme={null} observal admin saml-config-delete --force --output json ``` JSON deletion requires `--force` or `--yes`. ## SCIM tokens ```bash theme={null} observal admin scim-tokens --output json observal admin scim-token-create --description 'Okta' --output json observal admin scim-token-revoke 11111111-1111-1111-1111-111111111111 --force --output json ``` List results contain metadata and token prefixes only. Creation returns the plaintext bearer token once. Treat that result as a secret. Revocation requires a complete UUID and prompts in human mode unless forced. ## Security events ```bash theme={null} observal admin security-events --limit 50 --offset 0 --output json observal admin security-events --type auth.login.failure --severity critical --output json observal admin security-events --actor alice@example.com --output json ``` Severity accepts `info`, `warning`, or `critical`. Limit accepts 1 through 1,000 and offset accepts zero or greater. JSON returns the server envelope with `events` and `total`. ## Audit log Query events: ```bash theme={null} observal admin audit-log --limit 100 --offset 0 --output json observal admin audit-log --actor alice@example.com --resource-type agent --output json observal admin audit-log --source cli --outcome success --output json observal admin audit-log --start-date 2026-08-01 --end-date 2026-08-31 --output json ``` Available filters are action, actor, resource type, sensitivity, outcome, source, start date, and end date. Source accepts `server` or `cli`. Limit accepts 1 through 500. Print CSV to stdout: ```bash theme={null} observal admin audit-log-export ``` Write CSV atomically: ```bash theme={null} observal admin audit-log-export --file audit.csv ``` Print JSON: ```bash theme={null} observal admin audit-log-export --output json ``` Write JSON atomically: ```bash theme={null} observal admin audit-log-export --output json --file audit.json ``` Existing files prompt in human mode. JSON mode fails with a conflict unless `--force` or `--yes` is provided. Audit exports can contain sensitive administrative data. ## Review queue Reviews are revision-based and use the `observal review` command group. Content is visible only to participants and authorized reviewers. ```bash theme={null} observal review list --output json observal review list --type mcp --output json observal review list --team-id 11111111-1111-1111-1111-111111111111 --output json observal review show 123 --output json observal review files 123 --output json observal review diff 123 --file SKILL.md --output json observal review checks 123 --output json ``` Review data can include submitted configuration. Handle it as potentially sensitive. Credentials are redacted in snapshots. Submit a verdict and publish. Approval never publishes by itself unless the effective policy sets automatic publishing: ```bash theme={null} observal review approve 123 --output json observal review request-changes 123 --body 'Please document the required variables' --output json observal review publish 123 --output json ``` Review policy sets required approvals (0 to 5 per type), self-approval, and automatic publishing. A teamspace may set one policy for team-visible items and another for its public namespace; neither can be looser than the organization policy for public items. Zero approvals with automatic publishing is the "auto approve" setting. ```bash theme={null} observal review policy show --output json observal review policy set required_approvals.mcp 2 --output json ``` Use `observal review --help` for threads, comments, reviewer requests, dismissal, and watch/mute. ### Automated checks (experimental) Experimental and off by default. A super admin or admin turns it on under Settings, Experimental (`experimental.review_checks`). While off, the checks tab, queue labels, settings and endpoints below are hidden, and automated checks neither run nor block publishing; the built-in secret, schema and skill folder checks still apply. Each revision runs built-in checks (secret scan, required fields, skill folder validity) plus configurable automated checks on the worker, within about 15 seconds of submission: | Check | Applies to | Reads or contacts | | - | - | - | | Harness config generation | agent | nothing external | | Script syntax and risky patterns | hook, skill | nothing external; never executes scripts | | Prompt template variables | prompt | nothing external | | High-entropy secret heuristic | all | nothing external | | Sandbox image pinned by digest | sandbox | nothing external | | Git source resolves | mcp, skill, hook, sandbox | the Git host | | Remote MCP server responds | mcp | the MCP URL (anonymous, public addresses only) | | Sandbox image exists | sandbox | the image registry (anonymous) | | Known-vulnerable dependencies | mcp, skill, hook, sandbox | Git host and osv.dev (package names and versions) | Checks that contact external hosts, other than the Git source check, are off by default. Every check is advisory by default; an admin or team owner can mark one as blocking. A blocking check that is queued, running, failed or errored holds publication, including automatic publication. A check that could not run reports `skipped` or `error`, never `pass`. Settings are scoped. Admins edit the organization settings (used for public items) with the gear button on the review queue, or `PUT /api/v1/admin/review-checks`. Team owners and admins edit their teamspace's settings (used for team-visible items) on the teamspace Settings tab, or `PUT /api/v1/teams/{id}/review-checks`. Changes apply to new revisions; `POST /api/v1/reviews/{number}/checks/rerun` re-queues an open review under the current settings. These checks are static and network lookups: they do not prove a component is safe or that it works. ## Exit codes | Code | Meaning | | - | - | | 3 | Authentication required or failed | | 4 | Administrator or reviewer permission denied | | 5 | User, token, setting, review, or configuration not found | | 6 | Ambiguous reference or existing export conflict | | 7 | Invalid role, filter, UUID, SAML input, reason, or missing non-interactive confirmation | | 8 | Rate limit reached | | 9 | Server, database, ClickHouse, Redis, or filesystem unavailable | | 10 | CLI and server version mismatch | ## Related * [`observal auth`](/cli/auth): inspect the active account and role * [`observal inbox`](/cli/inbox): review and security notifications * [`observal ops`](/cli/ops): sessions, telemetry, logs, and insights # observal agent Source: https://docs.observal.io/cli/agent Create, compose, publish, install, and govern agents. An agent bundles MCP servers, skills, hooks, prompts, and sandboxes into one versioned Registry object. Canonical identities use `namespace/slug`. Commands also accept UUIDs, unambiguous bare names, aliases, and row numbers from the latest Agent list. ## Commands | Command | Purpose | | - | - | | `create` | Create from flags, JSON, or an interactive wizard | | `bulk-create` | Validate or create multiple agents from JSON | | `list` | List approved visible agents | | `my` | List agents owned or co-authored by the user | | `show` | Show an agent and its composition, including visible fork provenance and public fork count | | `fork` | Copy an approved agent release into an independent draft | | `install` | Generate an installation config without writing files | | `pull` | Generate and write a complete harness installation | | `archive` | Archive an agent | | `delete` | Compatibility alias for `archive` | | `unarchive` | Restore an archived agent | | `init` | Create a local `observal-agent.yaml` | | `add` | Add a Registry component UUID to local YAML | | `build` | Validate local composition and target scope | | `publish` | Create, update, save, or submit an agent | | `release` | Publish a reviewed version bump | | `versions` | List version history | | `outdated` | Show which pinned components have newer approved releases | | `transfer-owner` | Transfer ownership | | `co-authors` | List, add, or remove co-authors | Every leaf command supports `--output table|json`. JSON success output contains no Rich text, prompt, banner, or spinner. JSON failures leave stdout empty and write one categorized object to stderr. ## Create Use complete flags for automation: ```bash theme={null} observal agent create \ --name reviewer \ --description 'Reviews changes' \ --prompt 'Review carefully.' \ --model claude-sonnet-4 \ --harness kiro \ --output json ``` Other modes: ```bash theme={null} observal agent create --from-file agent.json --output json observal agent create ``` The no-flag form is interactive and cannot run in JSON mode. Flag mode requires `--name` and either `--prompt` or `--prompt-file`. Versions must be semantic versions and every repeated `--harness` must be registered. `--team HANDLE --visibility team` creates a private teamspace agent. `--visibility team` requires `--team`. JSON returns the direct server Agent object. ## Bulk create Input may be an array or an object containing an `agents` array: ```json theme={null} { "agents": [ { "name": "reviewer", "version": "1.0.0", "owner": "alice", "prompt": "Review carefully.", "model_name": "claude-sonnet-4", "components": [] } ] } ``` ```bash theme={null} observal agent bulk-create --from-file agents.json --dry-run --output json observal agent bulk-create --from-file agents.json --yes --output json ``` Input contains 1-50 agents. The CLI rejects malformed fields and duplicate canonical names before HTTP, and the server applies the same duplicate rules. JSON mode requires either `--dry-run` or `--yes`. Results include per-agent statuses, summary counts, and `partial`. Item errors set `partial: true` and exit with code 11; skips alone remain successful. ## List, my, and show ```bash theme={null} observal agent list --search 'incident response' --output json observal agent list --namespace alice --page 2 --limit 20 --output json observal agent list --team platform --output json observal agent my --output json observal agent show alice/reviewer --output json ``` `list` JSON is paginated: ```json theme={null} { "items": [], "total": 0, "page": 1, "page_size": 50 } ``` The most recent `list` or `my` result is cached for row-number references. Empty results clear stale Agent rows. `--interactive` is human-only and cannot be combined with JSON. `my` returns the standard `items`, `total`, `page`, and `page_size` envelope, including pending, approved, rejected, and archived agents. Empty results use `items: []`. `show` returns the direct Agent object, including component links and success criteria. ## Fork an approved agent ```bash theme={null} observal agent fork acme/pr-reviewer --name my-reviewer --output json observal agent fork acme/pr-reviewer --version 2.0.0 --new-version 0.1.0 --team payments --visibility team --output json observal agent fork acme/pr-reviewer --dir ./my-reviewer --output json ``` `SOURCE` accepts a UUID, `namespace/slug`, an alias, or a cached row number. Only a currently approved listing and an approved source version can be forked. The default is the newest stable approved version. A fork begins as **your draft** in a personal or permitted team namespace; it keeps approved component references, not copies of the components. A private source may be forked only into the same private teamspace. A name collision returns conflict rather than changing an existing agent. A disabled fork setting returns permission denied; per-user rate limiting returns exit code 8. `--dir` creates a new `observal-agent.yaml` containing the fork ID and editable fields, including exact component pins, `model_config_json`, and `external_mcps`. It refuses to overwrite an existing file *before* sending the fork request. JSON returns the direct draft object, plus `yaml_path` when provided. To edit this fork without creating a second listing: ```bash theme={null} observal agent build --dir ./my-reviewer --output json observal agent publish --update --dir ./my-reviewer --output json observal agent publish --submit AGENT_UUID --output json ``` `--update` uses the scaffold's `agent_id` to target the draft exactly, even though drafts do not appear in the public search. The submit command uses the ID returned by fork. A draft requires review before appearing in public fork counts or lists. `show` displays **Source unavailable** instead of saved source identity when access to the original has been lost; never infer provenance from a cached reference. ## Generate installation config `install` asks the server to generate config but does not write it: ```bash theme={null} observal agent install alice/reviewer --harness kiro --output json ``` JSON returns the complete server installation result. Legacy `--raw` prints only `config_snippet`; new automation should use the shared output option. Use [`observal agent pull`](/cli/pull) to write and track the generated installation. ## Archive, delete, and restore ```bash theme={null} observal agent archive alice/reviewer --yes --output json observal agent delete alice/reviewer --yes --output json observal agent unarchive alice/reviewer --yes --output json ``` `delete` is an alias for the same reversible archive operation. JSON mode never prompts and requires `--yes`. JSON returns the direct updated Agent object. ## Local authoring workflow ### Initialize ```bash theme={null} observal agent init \ --dir ./reviewer \ --name reviewer \ --description 'Reviews changes' \ --prompt-file ./PROMPT.md \ --model claude-sonnet-4 \ --harness kiro \ --output json ``` The no-flag form is interactive. JSON mode requires complete flags and refuses to overwrite an existing definition. Writes are atomic. #### From a working session ```bash theme={null} observal agent init --from-capabilities --name pr-review-flow --dir ./pr-review-flow --output json observal agent init --from-capabilities --since 3d --name data-helper --output json ``` `--from-capabilities` pre-fills `components` and `supported_harnesses` from the resources recorded in the capability lock for the current directory (see [`observal discover`](/cli/discover)): everything loaded with `discover use` and every component installed here in the last 24 hours (`--since` accepts `45m`, `2h`, `3d`, `1w`). In JSON mode a default description and prompt are filled in when omitted. Agents cannot nest other agents, so a pulled agent is reported under `from_capabilities.skipped_agents` rather than added. Exit code `5` when nothing was used in the window. JSON returns: ```json theme={null} { "path": "/work/reviewer/observal-agent.yaml", "agent": { "name": "reviewer", "version": "1.0.0" } } ``` ### Add components Find a component in Registry JSON, then copy its UUID: ```bash theme={null} observal registry skill list --search review --output json observal agent add skill 22222222-2222-2222-2222-222222222222 --dir ./reviewer --output json ``` Valid types are `mcp`, `skill`, `hook`, `prompt`, and `sandbox`. Duplicate type and UUID pairs return conflict exit code 6. Invalid types or IDs return validation exit code 7. Pin an exact component release with `--version`: ```bash theme={null} observal agent add mcp 33333333-3333-3333-3333-333333333333 --version 1.4.2 --dir ./reviewer --output json ``` This writes `version: 1.4.2` on the component entry. Without a version, the Agent keeps the version its current release pins, or pins the latest approved release when it first gains the component. Versions are matched exactly; `latest` and ranges are not accepted. A release pins approved versions only; a draft may also pin a component release still in review, but only one you can read (your own component, or as an admin or reviewer). JSON returns the YAML path and added component. ### Build ```bash theme={null} observal agent build --dir ./reviewer --output json observal agent build --dir ./reviewer --team platform --visibility team --output json ``` Build verifies every component, checks that any pinned component `version` exists, and validates whether private components are visible to the target owner. A successful JSON result contains `valid`, `agent`, `components`, and `issues`. Invalid composition exits with code 7 and leaves JSON stdout empty; the stderr error includes the same repairable data under `error.result`. ### Publish ```bash theme={null} observal agent publish --dir ./reviewer --output json observal agent publish --dir ./reviewer --draft --output json observal agent publish --submit alice/reviewer --output json observal agent publish --dir ./reviewer --update --bump minor --output json ``` `--draft` and `--submit` are mutually exclusive. Scope changes cannot be combined with `--update`; use ownership transfer or a separate visibility operation. Non-interactive updates may use `--bump patch|minor|major`. `--update` edits the latest version in place, so it only works while that version is a draft, pending, or rejected. An approved version is immutable: installs and locks depend on it. The server answers conflict exit code 6 and points to `observal agent release`, which publishes a new version for review. JSON returns the direct created, saved, submitted, or updated Agent object. ## Release and versions ```bash theme={null} observal agent release alice/reviewer --bump patch --dir ./reviewer --output json observal agent versions alice/reviewer --page 1 --page-size 50 --output json ``` Release obtains the server's semantic-version suggestion, submits the complete YAML snapshot, then updates local YAML atomically only after the server accepts the release. A failed server request leaves the local version unchanged. Every release pins each component to an exact version, and those pins are what every pull of the release installs. A release keeps the pins of the Agent's current release, so components never move unless the author asks: * give a component a `version` in the YAML to pin that release; * pass `--refresh-components` to move every other component to its latest approved release. ```bash theme={null} observal agent release alice/reviewer --bump minor --dir ./reviewer --refresh-components --output json ``` Approval freezes the release's lock. The review gate checks that each pinned component version is approved. ## Check component pins ```bash theme={null} observal agent outdated alice/reviewer observal agent outdated alice/reviewer --version 1.2.0 --output json ``` Lists each component the Agent version pins with its pinned version, its latest approved release, and a status: `current`, `outdated`, or `unlocked` (Agent versions released before pinning). Archived components are marked. Defaults to the latest Agent version. JSON returns `agent_id`, `qualified_name`, `version`, `components`, and a `summary` with `total`, `outdated`, `unlocked`, and `archived` counts. Consumers keep installing the pins; this is the author's cue to release a refreshed version. Versions JSON returns the direct paginated server object. Page size is 1 through 100. ## Transfer ownership ```bash theme={null} observal agent transfer-owner alice/reviewer bob --yes --output json ``` The username may optionally begin with `@`. JSON mode requires `--yes` and returns the direct server result. ## Co-authors Co-authors can edit and publish the same Agent: ```bash theme={null} observal agent co-authors list alice/reviewer --output json observal agent co-authors add alice/reviewer dev@example.com --output json observal agent co-authors add alice/reviewer @dev --output json observal agent co-authors remove alice/reviewer 550e8400-e29b-41d4-a716-446655440000 --output json ``` List returns the standard list envelope. Add returns the added user. Remove requires the user UUID returned by list and returns the direct deletion result. ## Pull ```bash theme={null} observal agent pull alice/reviewer --harness kiro --no-prompt --output json ``` Pull writes harness files, records the exact Agent and component versions in `observal.lock` and the local lockfile, and reports every file and setup action. Later pulls keep the locked version until `--upgrade` or `--version`. See the [Pull reference](/cli/pull) for pinning, strict mode, path, secret, merge, dry-run, and JSON behavior. ## Exit codes Common Agent failures use: | Code | Meaning | | - | - | | 3 | Authentication required or failed | | 4 | Permission denied | | 5 | Agent, component, or local definition not found | | 6 | Existing definition, duplicate component, or unsafe config merge conflict | | 7 | Invalid name, version, harness, component, scope, or command combination | | 8 | Server rate limit | | 9 | Server, filesystem, lockfile, generated config, or setup dependency unavailable | | 10 | CLI and server version mismatch | | 11 | Bulk operation returned one or more item errors | ## Related * [`observal agent pull`](/cli/pull): install into a harness * [`observal registry`](/cli/registry): manage Agent components * [`observal registry models`](/cli/models): inspect exact harness model IDs # observal api Source: https://docs.observal.io/cli/api Call an authenticated JSON endpoint when no dedicated high-level command exists. ## Examples ```bash theme={null} observal api GET /api/v1/teams --output json observal api GET /api/v1/agents --param tag=review --param tag=security --output json observal api POST /api/v1/teams --from-file team.json --output json cat team.json | observal api POST /api/v1/teams --output json ``` Methods are `GET`, `POST`, `PUT`, `PATCH`, and `DELETE`. Paths must be canonical relative `/api/v1/` paths; the leading slash is optional, so `api/v1/teams` works too. Full URLs, encoded or plain traversal segments, fragments, and inline query strings are rejected. Use repeatable `--param KEY=VALUE` options for query parameters; repeated keys preserve their original order. Git Bash on Windows rewrites an argument that starts with `/` into a Windows path, so `/api/v1/teams` reaches the CLI as `C:/Program Files/Git/api/v1/teams`. The CLI recognizes this, calls `/api/v1/teams`, and prints a note on standard error. Writing `api/v1/teams` without the leading slash avoids the rewrite. The command uses the configured bearer token. It does not accept arbitrary authorization headers, so it cannot forward credentials to another host. ## Request bodies `POST`, `PUT`, `PATCH`, and `DELETE` accept any single JSON value from `--from-file` or standard input: objects, arrays, strings, numbers, booleans, and explicit `null`. A file takes precedence when both are present. `GET` rejects every request body, including `null`. ## Output Table mode renders arbitrary objects as field and value rows. JSON mode preserves the endpoint response exactly, including top-level arrays. This raw behavior is intentional and is the exception to dedicated list commands, which use the standard list envelope. ## Errors and retries The command uses the shared categorized error contract and preserves server request IDs. Automatic transient retries apply only to `GET`. After an uncertain mutation failure, read endpoint state before retrying. See [Mutation retries and idempotency](/cli/idempotency). Prefer dedicated commands when they exist because they provide stronger validation, safer confirmation, and domain-specific output. # observal auth Source: https://docs.observal.io/cli/auth Authentication and account management. ## Commands | Command | Description | | - | - | | `observal auth login` | Authenticate with credentials or browser SSO | | `observal auth logout` | Sign out every session for this account (CLI, hooks, web) and clear local credentials | | `observal auth whoami` | Show the authenticated user | | `observal auth status` | Check authenticated server and local outbox health | | `observal auth change-password` | Change the current user's password; signs out every other session, this CLI stays signed in | | `observal auth set-username` | Set or update the registry namespace username | ## Login ```bash theme={null} observal auth login observal auth login --server https://observal.example.com --email alice observal auth login --sso observal auth login --sso --provider google ``` Login accepts `server`, `email`, `password`, `name`, `sso`, `saml`, `provider`, `output`, and `no-setup` options. Prefer `OBSERVAL_PASSWORD` or `OBSERVAL_PASSWORD_FILE` over the password option so the secret does not enter shell history or process arguments. Human login always asks for the server URL unless `--server` is supplied; leave the prompt blank to use `http://localhost`. On a fresh server, provide email, name, and a password to create the first administrator. JSON mode never prompts, uses the configured server or local default, and requires complete credential inputs. Successful human login first installs the Claude Code session hooks when Claude Code is already present (a `~/.claude` directory or a `claude` binary on `PATH`), then synchronizes the bundled skills, creates the initial layer snapshot, and runs doctor. A Claude Code settings file that is not valid JSON is left untouched with a note to run `observal doctor patch --harness claude-code`. Select `no-setup` to skip the snapshot and doctor. JSON mode skips those post-login steps. `provider` picks the browser SSO provider: `oidc`, `saml`, `google` or `github`. It implies `sso`, and the CLI rejects a provider the server has not enabled, or `saml` combined with a different `provider`. A bare `sso` uses the only enabled provider, or asks which one when the server enables several. JSON mode and non-interactive input never ask, so a bare `sso` there falls back to the server default (OIDC, then SAML); pass `provider` to choose. Every CLI invocation also computes a SHA-256 hash for each installed Observal-managed skill tree. A mismatched tree is replaced completely from the packaged bundle, including references and scripts, so local edits and stale extra files do not survive. Skill directories outside the six bundled Observal names are untouched. Credential JSON login emits one safe object and never includes tokens or passwords: ```bash theme={null} OBSERVAL_PASSWORD_FILE=/run/secrets/observal-password \ observal auth login \ --server https://observal.example.com \ --email alice \ --output json ``` Browser SSO in JSON mode is a JSON Lines stream. The first event contains the verification URL and user code. The final event confirms authentication: ```bash theme={null} observal auth login --sso --output json ``` ## Logout ```bash theme={null} observal auth logout observal auth logout --output json ``` Logout always removes local tokens when the local configuration is readable. Remote revocation is best effort and is reported separately in JSON. Use `observal doctor cleanup` to remove Observal-managed harness hooks. A logout from any device, including the web UI, also signs out the hooks on every other machine. When a hook's token refresh is rejected, the hooks stop retrying until the next `observal auth login`, and every other `observal` command prints a notice that sessions are not being sent. ## Current user ```bash theme={null} observal auth whoami observal auth whoami --output json ``` The JSON form returns the server user object directly. ## Status ```bash theme={null} observal auth status observal auth status --output json ``` Status reports the server URL, authentication state, health latency, and local telemetry outbox. It returns exit code 3 when authentication is absent and exit code 9 when the configured server is unreachable. ## Change password ```bash theme={null} observal auth change-password ``` Both modes read `OBSERVAL_CURRENT_PASSWORD` and `OBSERVAL_NEW_PASSWORD`, including their corresponding `_FILE` forms. Human mode prompts for missing values; JSON mode requires both values and never prompts. ```bash theme={null} OBSERVAL_CURRENT_PASSWORD_FILE=/run/secrets/current-password \ OBSERVAL_NEW_PASSWORD_FILE=/run/secrets/new-password \ observal auth change-password --output json ``` Passwords require at least 12 characters, one uppercase letter, one number, and one special character. ## Set username ```bash theme={null} observal auth set-username alice observal auth set-username alice --output json ``` The username is the user's registry namespace. It must follow the namespace rules shown by command help. The server prevents changes that conflict with another namespace or published registry ownership. ## Environment variables | Variable | Purpose | | - | - | | `OBSERVAL_SERVER_URL` | Default server URL | | `OBSERVAL_ACCESS_TOKEN` | Pre-authenticate commands without login | | `OBSERVAL_PASSWORD` | Credential or bootstrap login password | | `OBSERVAL_CURRENT_PASSWORD` | Current password for JSON password changes | | `OBSERVAL_NEW_PASSWORD` | New password for JSON password changes and mandatory login changes | | `OBSERVAL_TIMEOUT` | Authenticated client timeout | Password and token variables support a corresponding `_FILE` form. Do not set a direct value and its file form together. Full list: [Environment variables](/reference/environment-variables). ## Related * [`observal config`](/cli/config): local configuration * [Self-hosting authentication](/self-hosting/authentication): server authentication and SSO # observal registry version Source: https://docs.observal.io/cli/component Publish and inspect versioned releases for MCP servers, skills, hooks, prompts, and sandboxes. ## Commands | Command | Description | | - | - | | `publish` | Publish a new component version for review | | `list` | List paginated version history | Valid component types are `mcp`, `skill`, `hook`, `prompt`, and `sandbox`. ## Publish ```bash theme={null} observal registry version publish mcp acme/server \ --version 2.0.0 \ --description "New authentication flow" \ --changelog "Breaking change" \ --output json ``` | Option | Description | | - | - | | `--version`, `-v` | Version to publish | | `--description`, `-d` | Required release description | | `--changelog` | Optional release notes | | `--harness` | Supported harness; repeatable | | `--extra` | Type-specific metadata as a JSON object | | `--output`, `-o` | Table or JSON output | Human mode fetches version suggestions and prompts when a version is omitted. JSON mode never prompts and therefore requires an explicit version. Suggestion failures retain their original authentication, rate-limit, unavailable, or version-mismatch category. The extra value must be a JSON object. Versions and harnesses are validated before publication. ## List ```bash theme={null} observal registry version list mcp acme/server --output json observal registry version list hook acme/guard --page 2 --page-size 100 --output json ``` | Option | Description | | - | - | | `--page` | Page number, starting at 1 | | `--page-size` | Items per page, from 1 through 200 | | `--output`, `-o` | Table or JSON output | JSON returns the direct paginated server object with `items`, `total`, `page`, and `page_size`. Empty pages preserve that shape. Human output shows version, review status, relative release date, and publisher. ## Exit codes | Code | Meaning | | - | - | | `0` | Publication or listing completed | | `2` | Invalid command syntax or pagination bounds | | `3` | Authentication required or failed | | `4` | Permission denied | | `5` | Component not found | | `6` | Version or state conflict | | `7` | Invalid component type, version, harness, or metadata | | `8` | Rate limit reached | | `9` | Registry unavailable | | `10` | CLI and server version mismatch | ## Related * [`observal registry`](/cli/registry): component management * [`observal outdated`](/cli/outdated): compare installed and latest versions # observal config Source: https://docs.observal.io/cli/config Manage local CLI configuration. Configuration lives in `~/.observal/config.json`; aliases live in `~/.observal/aliases.json`. Both files are written atomically with mode `0600`. ## Commands | Command | Description | | - | - | | `observal config show` | Show effective configuration without exposing credentials | | `observal config set` | Set a validated user-managed value | | `observal config path` | Print the configuration file path | | `observal config alias` | Set or remove a local reference alias | | `observal config aliases` | List local aliases | Every command supports explicit table and JSON output. Human-readable table output is the default. `config path` preserves a bare path in table mode for shell composition. ## Show configuration ```bash theme={null} observal config show observal config show --output json ``` Token values and fragments are never displayed. Output reports only whether access, refresh, or harness-hook credentials are configured. Environment overrides are included because the command shows effective configuration. ## Set configuration ```bash theme={null} observal config set server_url https://observal.example.com observal config set timeout 60 --output json observal config set update_check false observal config set update_check_interval 86400 observal config set update_check_repo Observal/Observal ``` Only these user-managed keys are accepted: | Key | Accepted value | | - | - | | `server_url` | HTTP or HTTPS URL without embedded credentials | | `timeout` | Positive integer seconds | | `update_check` | `true`, `false`, `yes`, `no`, `on`, `off`, `1`, or `0` | | `update_check_interval` | Integer seconds, minimum 60 | | `update_check_repo` | `owner/repository` or an empty string | Authentication and identity fields are managed by `observal auth` and cannot be set here. The removed `output` and `color` settings had no runtime consumers. Select JSON explicitly for each command. JSON output distinguishes the persisted value from the effective value, which may be overridden by an environment variable. ## Configuration path ```bash theme={null} observal config path observal config path --output json ``` Table mode prints only the absolute path. JSON includes the path and whether the file currently exists. ## Set or remove an alias ```bash theme={null} observal config alias reviewer alice/reviewer observal config alias reviewer alice/reviewer --output json observal config alias reviewer ``` Alias names must start with a letter and may contain letters, numbers, dots, underscores, and hyphens, up to 64 characters. Targets may be UUIDs, canonical `namespace/slug` identities, names, or other references accepted by the destination command. Omitting the target removes the alias. Removing an already absent alias is an idempotent success with `changed` set to `false` in JSON. ## List aliases ```bash theme={null} observal config aliases observal config aliases --output json ``` JSON returns an `items` array and `total`. Empty output preserves the same shape. ## Environment overrides Environment values take precedence over persisted values for the current invocation. `config set` changes the file but does not replace an active environment override. See [Environment variables](/reference/environment-variables) for supported overrides. ## Related * [Config files](/reference/config-files): local file schemas and permissions * [`observal auth`](/cli/auth): authentication-managed values # observal discover Source: https://docs.observal.io/cli/discover Find approved registry resources for the task at hand and use them in the current session, without building an Agent first. Discovery searches every kind at once (agents, MCP servers, skills, hooks, prompts, sandboxes), ranks by relevance, and tells you separately whether each result is approved, whether it works in your harness, and whether it can be used right now or only after a restart. For signed-in users, results are scoped to personal listings they own and listings in every teamspace they currently belong to (including public teamspace listings); other people's public listings are not returned. Global reviewers and admins retain their review/admin scope. Anonymous public discovery is unchanged when the public registry is enabled. The bundled `observal` skill tells coding assistants to run this before reinventing something or declaring a capability unavailable. ## Search ```bash theme={null} observal discover search review a pull request for authentication bugs observal discover search 'query postgres' --type mcp --output json observal discover search generate tests --harness pi observal discover search release notes --include-unapproved ``` Pass the task as one shell-escaped argument (single-quote it and escape any embedded `'` as `'\''`). Several bare words are still joined into one query, but the text comes from the user and must never be spliced into a shell command unquoted. | Option | Description | | - | - | | `--type`, `-t` | Restrict to `agent`, `mcp`, `skill`, `hook`, `prompt`, or `sandbox` | | `--harness`, `-i` | Only resources that explicitly list this harness as supported. Resources that declare no harnesses are left out (support is unknown, not implied); drop the flag to see them. Remote A2A agents are kept, since they run elsewhere. | | `--limit`, `-n` | Number of results, 1 through 25 (default 5) | | `--include-unapproved` | Also return your own pending or rejected submissions | | `--output`, `-o` | Table or JSON output | JSON returns `query`, `harness`, `count`, and `results[]`. Important result fields: | Field | Meaning | | - | - | | `identifier` | Permanent `urn:air:::` used by `inspect` and `use` | | `score` | Relevance to the query, 0 to 100. Never approval or trust. | | `matchedOn` | The query words that matched | | `obs:kind` | `agent`, `mcp`, `skill`, `hook`, `prompt`, `sandbox` | | `obs:nativeRef` | `namespace/slug@version`, usable with the registry and agent commands | | `obs:approval` | `approved`, `pending`, `rejected`, `archived`, `draft` | | `obs:availability` | `now`, `next-session`, `explicit-install`, `not-approved`, `archived`, `unsupported-in-harness` | | `obs:supportedHarnesses` | Harnesses the publisher declared; empty means support is unknown (not unrestricted). Policy: `--harness` search needs positive evidence, so it leaves out resources that declare no harnesses; `inspect` and `use` only refuse a resource that declares other harnesses, so a resource that declares none stays usable. | | `obs:artifactDigest` | SHA-256 of the exact artifact bytes | An empty `results` array is a successful answer: nothing matched. Try fewer or different words before concluding the resource does not exist. ## Inspect ```bash theme={null} observal discover inspect urn:air:observal.acme.com:skill:5f2c... --output json ``` Returns the complete entry, including `representativeQueries` (what the resource is good for), `capabilities`, the versioned artifact `url`, and its digest. The predecessor `urn:ai:` prefix is accepted. ## Use ```bash theme={null} observal discover use urn:air:observal.acme.com:skill:5f2c... observal discover use urn:air:observal.acme.com:mcp:9a1b... --harness kiro --output json ``` | Option | Description | | - | - | | `--harness`, `-i` | Harness this session runs in. Detected from `OBSERVAL_HARNESS` or the only installed harness when omitted. | | `--max-chars` | Cap on printed content (default 32000); longer artifacts are truncated with a pointer to the full URL | | `--yes`, `-y` | Load a resource that is not approved (your own drafts) | | `--output`, `-o` | Table or JSON output | What happens depends on the kind: * **Skills and prompts** are text. The exact approved version is fetched from its permanent URL and printed (JSON: `content`, `activated: true`, `mode: context`). The use is recorded in the capability lock. * **MCP servers, agents, and hooks** change your setup. Nothing is written; the response carries `next_step`, the existing install command (`observal registry mcp install …`, `observal agent pull …`), which confirms before touching harness config and takes effect after a restart. JSON: `activated: false`, `mode: next-session`. * **Sandboxes** have no install command; they reach a machine only inside an Agent. `next_step` is `observal agent add sandbox [--version V]`, run in an Agent directory (`observal agent init`); `reason` says to then `observal agent publish` and `observal agent pull --harness `. JSON: `activated: false`, `mode: next-session`. Unapproved resources are refused without `--yes`. A resource whose `obs:supportedHarnesses` excludes the detected harness is refused; pass `--harness` to override the detection. ## The capability lock Every activation appends one line to `~/.observal/capability_lock.jsonl`: timestamp, harness, working directory, identifier, kind, version, digest, and mode. `agent pull` and the component install commands append too. Session upload (`session_push` hooks and `observal reconcile`) attaches the lines that fall inside a session's harness, directory, and time window as `capabilities_used`, which is how Observal knows which resources a session relied on. Nothing secret is stored. Entries older than 30 days are pruned. Related: [`observal agent init --from-capabilities`](/cli/agent) turns the resources used in a directory into a draft Agent. ## Exit codes Follows the shared CLI error contract: `4` permission (unapproved resource without `--yes`), `5` not found, `7` validation (unknown type, harness, or identifier). # observal doctor Source: https://docs.observal.io/cli/doctor Diagnose local Observal state, repair managed telemetry instrumentation, remove managed instrumentation, and work with redacted support bundles. All five workflows support `--output table|json`. JSON mode never prompts or emits human banners. ## Commands | Command | Purpose | | - | - | | `doctor` | Diagnose configuration, Registry metadata, skills, and harness telemetry | | `doctor patch` | Install or update Observal-managed telemetry instrumentation | | `doctor cleanup` | Remove Observal-managed telemetry instrumentation | | `doctor support bundle` | Generate a redacted diagnostic archive | | `doctor support inspect` | Inspect a diagnostic archive without extracting it | ## Diagnose ```bash theme={null} observal doctor --output json ``` Doctor checks: * Local authentication configuration and server health * Registry lockfile metadata against the active Registry * Managed hooks, plugins, or extensions for all registered harnesses * UUID-attributed Kiro Agent hooks * Bundled Observal skill installation JSON diagnosis exits zero when the checks ran successfully. Health is reported through `healthy`, `issues`, and `warnings`: ```json theme={null} { "healthy": false, "issues": ["Cannot reach the configured server"], "warnings": ["Cursor session push hooks are not installed"], "lockfile_changes": [], "skill_missing": [], "fix_attempted": false, "patch": null } ``` Human mode retains health-check behavior: unresolved issues exit nonzero. Warnings alone remain successful. Apply fixable warnings and canonical lockfile metadata without prompting: ```bash theme={null} observal doctor --yes --output json ``` Installed version pins are not changed. Only Observal-managed telemetry entries are updated. ## Patch Preview every registered harness: ```bash theme={null} observal doctor patch --all-harnesses --dry-run --output json ``` Patch selected harnesses: ```bash theme={null} observal doctor patch --harness claude-code --harness kiro --output json ``` Exactly one target mode is required: `--all-harnesses` or one or more `--harness` options. JSON returns one result per harness: ```json theme={null} { "action": "patch", "dry_run": false, "changed": true, "targets": [ {"harness": "kiro", "changed": true} ] } ``` Patch is idempotent. It preserves unrelated hooks and configuration. Configuration writes are atomic. Invalid or unreadable harness files fail loudly rather than being replaced. For Pi, patch installs the bundled TypeScript extension at `~/.pi/agent/extensions/observal.ts` and records the CLI version it came from in an adjacent `.observal-extension.json`. If `npm:observal-pi` is registered in `~/.pi/agent/settings.json`, that registration wins: patch installs nothing locally, and removes a local copy it recognises as its own so Pi does not load the extension twice. A local file Observal did not write is never touched. MCP commands and remote URLs are never wrapped or rewritten. ## Cleanup Preview cleanup: ```bash theme={null} observal doctor cleanup --dry-run --output json ``` Remove instrumentation from one harness: ```bash theme={null} observal doctor cleanup --harness goose --yes --output json ``` Remove instrumentation from all registered harnesses except selected entries: ```bash theme={null} observal doctor cleanup --exclude kiro --yes --output json ``` Cleanup removes only Observal-managed hooks, plugins, extensions, and legacy telemetry settings. User-owned hooks remain. Human mode confirms before writing unless `--yes` is present. JSON cleanup requires `--yes` unless it is a dry run. Unknown harnesses, conflicting selections, malformed configuration, and write failures are surfaced before success is reported. ## Support bundles ```bash theme={null} observal doctor support bundle --output json observal doctor support inspect ./observal-support.tar.gz --output json ``` See [Support bundles](/cli/support) for archive contents, redaction, offline behavior, and inspection limits. ## Exit codes | Code | Meaning | | - | - | | 3 | Authentication is required for patching | | 5 | Support bundle or requested bundle file not found | | 6 | Output archive already exists | | 7 | Invalid harness selection, malformed harness file, invalid bundle, or missing confirmation | | 9 | Server, collector, or filesystem unavailable | ## Related * [`observal scan`](/cli/scan): read-only harness inventory * [`observal agent pull`](/cli/pull): install a complete Agent * [Session tracking](/core-concepts/session-tracking): telemetry delivery architecture # Mutation retries and idempotency Source: https://docs.observal.io/cli/idempotency Agents must distinguish a rejected mutation from a mutation whose result is unknown. ## Automatic retry contract The shared CLI client automatically retries authenticated `GET` requests on HTTP 429, 503, and 504 responses, honoring `Retry-After` when present. It attempts token refresh once after a 401 response. `POST`, `PUT`, `PATCH`, and `DELETE` requests are sent once. The CLI does not retry transient mutation responses because the server may have applied the change before the connection failed. ## Agent workflow 1. Read current state and capture UUIDs, canonical names, versions, or status. 2. Send one mutation. 3. If the command succeeds, inspect the returned state and perform the smallest verification read when needed. 4. If validation, permission, or conflict is returned, correct that state rather than retrying unchanged. 5. If timeout, connection failure, 503, or 504 occurs after a mutation was sent, treat the result as unknown. Read current state before deciding whether to retry. ## Mutation families | Mutation | Verification before retry | | - | - | | Create Agent or component | List or show by canonical `namespace/slug`; do not create again if it exists | | Publish or release version | Read version history and review status | | Update visibility, role, or settings | Read the specific resource and compare the desired field | | Approve or reject a request | Read request status by UUID | | Archive, restore, revoke, or delete | Show the resource or confirm not-found where deletion is expected | | Install or pull | Inspect returned files and run `scan` for the target harness | | Bulk submit | Re-run only after reviewing per-entry results; existing identities return conflicts and are reported as skipped | | Server upgrade, rollback, or migration | Read service version, migration validation, and destination state before any retry | ## Conflict behavior A conflict is deterministic information, not a transient failure. Common decisions include: * Use `qualified_name` when a bare name is ambiguous. * Use update only when the user requested an in-place change. * Use release or version publish when the user requested a new version. * Treat an already-created bulk component as skipped after verifying its identity. * Do not bypass edit locks or ownership conflicts. The `observal api` escape hatch follows the same retry rules because it uses the shared authenticated client. # observal inbox Source: https://docs.observal.io/cli/inbox View and update the signed-in user's work and event feed. Inbox items cover reviews, decisions, teamspace activity, update notices, completed insights, and system notices. The Inbox is self-only. There is no option for reading another user's items. ## List items The direct and explicit list forms are equivalent: ```bash theme={null} observal inbox --output json observal inbox list --state open --action-required --output json observal inbox list --subject-type mcp --search postgres --sort oldest --output json ``` | Option | Description | | - | - | | `--state`, `-s` | Filter by `open`, `done`, or `dismissed` | | `--kind`, `-k` | Filter by event kind | | `--action-required` / `--no-action-required` | Filter by whether the item requires action | | `--unread` / `--read` | Filter by read state | | `--subject-type` | Filter by the related object type, such as `agent`, `mcp`, `skill`, `team`, or `insight_report` | | `--search`, `-q` | Search titles, bodies, namespaces, and slugs | | `--sort` | Sort by `newest` or `oldest` | | `--page`, `-p` | Select a page, starting at 1 | | `--page-size` | Return 1 through 100 items | | `--output`, `-o` | Table or JSON output | Supported kinds are: * `review_requested` * `review_approved` * `review_rejected` * `review_comment` * `change_requested` * `team_join_requested` * `team_join_decided` * `team_created_pending` * `ownership_transfer` * `update_available` * `insight_ready` * `system_notice` JSON returns the direct paginated server object: ```json theme={null} { "items": [], "total": 0, "page": 1, "page_size": 25 } ``` An empty `items` array is successful. Continue with `--page N` while the returned page range is below `total`. ## Count items ```bash theme={null} observal inbox count --output json observal inbox count --facets --facet-state open --output json ``` The basic result includes `unread`, `action_required`, `open`, `done`, and `dismissed`. `--facets` also returns `by_kind` and `by_subject_type`; `--facet-state` restricts those breakdowns and requires `--facets`. ## Show an item ```bash theme={null} observal inbox show 11111111-1111-1111-1111-111111111111 --output json ``` The detail result includes the item's body, related subject, exact action URL or command, and append-only history. The CLI displays an action command but never runs it automatically. ## Update one item Every mutation supports table and JSON output: ```bash theme={null} observal inbox read 11111111-1111-1111-1111-111111111111 --output json observal inbox unread 11111111-1111-1111-1111-111111111111 --output json observal inbox done 11111111-1111-1111-1111-111111111111 --output json observal inbox dismiss 11111111-1111-1111-1111-111111111111 --output json observal inbox reopen 11111111-1111-1111-1111-111111111111 --output json ``` Read state and lifecycle state are separate: * `read` and `unread` change only whether the item has been seen. * `done` resolves completed work. * `dismiss` resolves the item without acting on it. * `reopen` returns a done or dismissed item to `open`. JSON returns the updated item directly. ## Mark a filtered set read ```bash theme={null} observal inbox read-all --kind update_available observal inbox read-all --state open --subject-type mcp --search postgres --yes --output json ``` `read-all` affects only unread items matching the supplied state, kind, action-required, subject-type, and search filters. Human mode asks for confirmation unless `--yes` is present. JSON mode never prompts and therefore requires `--yes`. JSON returns: ```json theme={null} { "updated": 3 } ``` ## Exit codes Invalid states, kinds, sort orders, filter lengths, item IDs, and non-interactive confirmation use validation exit code 7. Authentication, permission, not-found, rate-limit, unavailable-service, and version failures preserve the shared CLI exit contract. In JSON mode, stdout contains only the successful result. Failures leave stdout empty and write one categorized error object to stderr. ## Related * [`observal outdated`](/cli/outdated): report installed updates to Inbox * [`observal registry`](/cli/registry): inspect Registry subjects referenced by Inbox items # Overview Source: https://docs.observal.io/cli/index Complete reference for the `observal` CLI. Every subcommand has its own page; this overview is the index. > **New to Observal?** Start with [Quickstart](/getting-started/quickstart) and come back here when you need a specific command. ## Command groups | Command | What it does | | - | - | | [`observal api`](/cli/api) | Call authenticated JSON endpoints without a dedicated command | | [`observal auth`](/cli/auth) | Authentication and account management | | [`observal config`](/cli/config) | Local CLI configuration, aliases | | [`observal discover`](/cli/discover) | Find approved resources for a task and use them in the current session | | [`observal scan`](/cli/scan) | Discover what's installed across your harnesses (read-only) | | [`observal share`](/cli/share) | Share and pull scoped, version-pinned repository Agent sets | | [`observal outdated`](/cli/outdated) | Compare installed agent and component versions with the active registry | | [`observal reconcile`](/cli/reconcile) | Backfill sessions missed by automatic telemetry delivery | | [`observal inbox`](/cli/inbox) | Read and update the signed-in user's work and event feed | | [`observal agent pull`](/cli/pull) | Install a published agent into a harness | | [`observal registry`](/cli/registry) | Publish and manage components (MCP / skill / hook / prompt / sandbox) | | [`observal registry recommend`](/cli/recommend) | Get personalized component recommendations | | [`observal registry version`](/cli/component) | Publish and inspect component versions | | [`observal registry models`](/cli/models) | Inspect packaged harness model catalogs | | [`observal agent`](/cli/agent) | Create, install, and manage agents | | [`observal team`](/cli/team) | Manage teamspaces, members, join requests, and invitations | | [`observal ops`](/cli/ops) | Observability and operations (sessions, telemetry, logs, insights) | | [`observal admin`](/cli/admin) | Core administration and submission review | | [`observal doctor support`](/cli/support) | Generate and inspect redacted diagnostic bundles | | [`observal doctor`](/cli/doctor) | Diagnose harness compatibility; `doctor patch` applies instrumentation | | [`observal server migrate`](/cli/migrate) | Export/import PostgreSQL registry (shallow copy) and ClickHouse telemetry (deep copy) | | [`observal self`](/cli/self) | Upgrade or downgrade the CLI | | [`observal registry prompt`](/cli/prompt) | Manage reusable prompts in the registry | | [`observal server`](/cli/server) | Manage the embedded server (start, stop, upgrade, rollback) | | [`observal registry skill`](/cli/skill) | Submit, browse, and install portable skill packages | ## Global options Any subcommand accepts these. | Option | Short | Description | | - | - | - | | `--version` | `-V` | Print the CLI version and exit | | `--verbose` | `-v` | Verbose output | | `--debug` | - | Debug-level logging (extremely verbose) | | `--help` | - | Show help for any command or subcommand | ## JSON list contract Every dedicated list command returns the same envelope: ```json theme={null} { "items": [], "total": 0, "page": 1, "page_size": 0 } ``` Paginated commands preserve server totals and requested pages. Unpaginated commands use page 1 and the returned item count as `page_size`. Detail and mutation commands return direct objects. `observal api` intentionally preserves raw endpoint JSON. ## Exit codes Consistent across all commands: | Code | Meaning | | - | - | | 0 | Success | | 1 | Unexpected or uncategorized failure | | 2 | Usage error | | 3 | Authentication required or failed | | 4 | Permission denied | | 5 | Resource not found | | 6 | Conflict with current state | | 7 | Validation failure | | 8 | Rate limit reached | | 9 | Network, service, or dependency unavailable | | 10 | CLI and server version mismatch | | 11 | Batch completed with one or more item errors | Errors identify the failed operation, resource, remediation, and server request ID when available. Repairable failures may include a safe structured `result`; internal details appear only with `--debug`. When JSON output is selected, errors are written to stderr as one JSON object and stdout remains clean. Startup migrations and update notices never write to JSON stdout. A nonfatal startup migration warning is a structured `warning` object on stderr. ```json theme={null} { "error": { "category": "not_found", "message": "The requested resource was not found.", "operation": "Show agent", "resource": "agent registry", "remediation": "Check the identifier or list available agents.", "request_id": "01J...", "http_status": 404, "exit_code": 5 } } ``` ## Non-interactive mode For scripts and CI, pair flags with environment variables: ```bash theme={null} export OBSERVAL_SERVER_URL=https://observal.your-company.internal export OBSERVAL_API_KEY='replace-with-your-key' observal ops traces --limit 100 --output json | jq ``` Full env var reference: [Environment variables](/reference/environment-variables). ## Output formats Read-heavy commands (`list`, `show`, `traces`, `spans`) support `--output`: ```bash theme={null} observal registry mcp list --output table # default, TTY-friendly observal registry mcp list --output json # machine-readable ``` ## Aliases IDs get long fast. Create shortcuts: ```bash theme={null} observal config alias my-mcp 498c17ac-1234-4567-89ab-cdef01234567 observal registry mcp show @my-mcp ``` See [`observal config`](/cli/config) for details. ## Next → [`observal auth`](/cli/auth): you'll need to log in first. # observal server migrate Source: https://docs.observal.io/cli/migrate Move PostgreSQL registry data and ClickHouse telemetry between Observal deployments. Migration uses the supplied database connections directly. Local shell and database access are the authorization boundary; the command does not authenticate against a configured Observal API. Install the optional dependency first: ```bash theme={null} pip install 'observal-cli[migrate]' ``` Keep connection URLs in environment variables or secret files managed by the shell. Source commands read `DATABASE_URL` and `CLICKHOUSE_URL`; target commands read `TARGET_DATABASE_URL` and `TARGET_CLICKHOUSE_URL`. Explicit URL options remain available when no secret is embedded. Do not paste credentials into shared shell history, logs, or issue reports. JSON results and categorized errors never echo a connection URL. ## Workflow 1. Export PostgreSQL. This creates a checksummed registry archive and a migration manifest. 2. Validate and import PostgreSQL on the target. 3. Export ClickHouse using the PostgreSQL migration manifest. 4. Validate and import ClickHouse on the target. PostgreSQL must be imported first so referenced users and agents exist before telemetry validation. ## PostgreSQL export ```bash theme={null} observal server migrate export \ --file registry.tar.gz \ --output json ``` `--file/-f` selects the archive destination. `--output/-o` always selects `table` or `json`. Existing destinations fail with a conflict instead of being overwritten. The archive and sidecar manifest are written atomically with owner-only permissions. A partial archive is not published after failure. Example JSON fields: ```json theme={null} { "archive": "registry.tar.gz", "manifest": "registry.manifest.json", "migration_id": "7b84e503-63af-4b89-a1cd-abf48f0452f3", "table_counts": {"users": 8, "agents": 21}, "total_rows": 342, "size_bytes": 1048576, "duration_seconds": 2.4 } ``` ## PostgreSQL validation and import ```bash theme={null} observal server migrate validate \ --archive registry.tar.gz \ --output json observal server migrate import \ --archive registry.tar.gz \ --output json ``` Validation checks archive structure and SHA-256 checksums. When a target URL is provided, it also compares table row counts. Checksum failure returns a categorized validation error. Row-count differences remain explicit result data. Import verifies checksums before insertion. Existing rows are skipped according to the migration service's idempotent import rules. The result contains per-table inserted and skipped counts plus warnings. ## ClickHouse export ClickHouse export requires the PostgreSQL sidecar manifest and a new destination directory: ```bash theme={null} observal server migrate export-telemetry \ --manifest registry.manifest.json \ --output-dir telemetry-export \ --output json ``` The destination must not already exist. This lets the exporter remove the complete directory after failure without touching pre-existing files. The directory and streamed Parquet files use restrictive permissions and atomic temporary files. The export covers active session, checkpoint, layer, audit, security, and webhook tables. Older sources may omit tables. Telemetry is divided into deterministic time-and-hash chunks so each ClickHouse query and Parquet file stays bounded independently of the total dataset size. Oversized or memory-limited chunks are split again automatically. `telemetry_manifest.json` records every chunk's range, shard, checksum, size, row count, and migration ID. ## ClickHouse validation and import ```bash theme={null} observal server migrate validate-telemetry \ --input-dir telemetry-export \ --output json observal server migrate import-telemetry \ --input-dir telemetry-export \ --output json ``` Telemetry validation checks: * Parquet checksums * Manifest row counts against target ClickHouse when supplied * Agent and user references against target PostgreSQL when supplied Checksum failure is fatal. Row-count differences and orphan groups are returned explicitly. Telemetry import is resumable per Parquet chunk. Successfully imported chunk IDs and checksums are recorded in `.import_state.json` in the input directory, so a retry skips only verified completed chunks rather than an entire table or month. Deterministic insert tokens protect ambiguous retries on current non-replicated MergeTree schemas. Imported project-keyed rows normalize to the deployment project `default`, and session summaries are rebuilt after event import so summaries reflect complete sessions rather than individual insert blocks. ## Human and JSON behavior All six leaves accept `--output table|json`. Human mode renders progress and summaries. JSON mode is finite, prompt-free, suppresses progress and warnings from stdout, and returns one result document. Failures leave stdout empty and emit one categorized error to stderr. Cleartext ClickHouse transport with credentials produces a human warning. JSON mode does not print a banner; operators should use `clickhouses://` for TLS. ## Exit codes | Code | Category | Typical migration cause | | - | - | - | | 2 | Usage | Missing required option or unsupported output mode | | 4 | Permission | Destination or source path is not accessible | | 5 | Not found | Archive, manifest, or input directory is missing | | 6 | Conflict | Archive or telemetry destination already exists | | 7 | Validation | Invalid archive, failed checksum, or missing phase prerequisite | | 9 | Unavailable | Optional dependency, database, network, or migration service failure | ## Recommended sequence ```bash theme={null} # Source observal server migrate export --file registry.tar.gz --output json observal server migrate export-telemetry \ --manifest registry.manifest.json \ --output-dir telemetry-export \ --output json # Target observal server migrate validate --archive registry.tar.gz --output json observal server migrate import --archive registry.tar.gz --output json observal server migrate validate-telemetry \ --input-dir telemetry-export \ --output json observal server migrate import-telemetry \ --input-dir telemetry-export \ --output json ``` # observal registry models Source: https://docs.observal.io/cli/models Display the model catalog packaged for registered harnesses. ## Synopsis ```bash theme={null} observal registry models [--harness ] [--output table|json] observal registry models list [--harness ] [--output table|json] ``` The direct and explicit list forms are equivalent. ## Options | Option | Description | | - | - | | `--harness ` | Filter to one registered harness | | `--output table` | Render a human-readable table; default | | `--output json` | Emit the complete catalog object | ## JSON schema ```json theme={null} { "models": [ { "harness": "pi", "model_id": "anthropic/claude-sonnet-4-6", "kind": "exact", "display_name": "Claude Sonnet 4.6" } ], "source": "harness-registry", "degraded": false } ``` An empty catalog keeps the same object with an empty `models` array. ## Data source The command reads harness model JSON files packaged under `observal_shared/harness_models/`. It does not contact the Observal server. An unknown harness is a usage error with exit code 2. JSON errors are written to stderr while stdout remains empty. ## Examples ```bash theme={null} observal registry models observal registry models --harness pi --output json observal registry models list --harness claude-code --output json ``` # observal ops Source: https://docs.observal.io/cli/ops Inspect current session telemetry, Registry usage, feedback, logs, and Agent insight reports. All 11 supported workflows provide table and JSON output. Finite commands return one JSON document. Streaming commands return JSON Lines, one compact object per update. The dead `ops metrics`, legacy `ops spans`, and synthetic `ops telemetry test` commands have been removed. Current trace detail comes from `ops traces --span` through the Sessions API. ## Commands | Command | Purpose | | - | - | | `top` | Show the most-downloaded MCPs or Agents | | `rate` | Submit Registry feedback | | `rate-update` | Update the caller's feedback | | `rate-delete` | Delete the caller's feedback | | `feedback` | Show aggregate and individual feedback | | `traces` | List current sessions or fetch their details | | `telemetry status` | Check server event counts and local outbox health | | `logs` | Follow local or remote development logs | | `insights list` | List Agent insight reports | | `insights show` | Show an insight report or section | | `insights generate` | Generate an insight report | ## Top Registry items ```bash theme={null} observal ops top --type mcp --output json observal ops top --type agent --output json ``` JSON returns the direct ranking array. MCP ranking is currently server-limited to five results; Agent ranking is currently server-limited to six. ## Feedback Submit feedback: ```bash theme={null} observal ops rate alice/postgres --type mcp --stars 5 --comment 'Reliable' --output json ``` Update feedback: ```bash theme={null} observal ops rate-update alice/postgres --type mcp --stars 4 --output json observal ops rate-update alice/postgres --type mcp --comment 'Updated review' --anonymous --output json ``` Delete feedback: ```bash theme={null} observal ops rate-delete alice/postgres --type mcp --yes --output json ``` View feedback: ```bash theme={null} observal ops feedback alice/postgres --type mcp --output json ``` Feedback types are `mcp`, `agent`, `skill`, `hook`, `prompt`, and `sandbox`. Stars accept 1 through 5 and comments accept at most 5,000 characters. Rate and update return the direct feedback object. Delete currently returns an empty object. JSON deletion requires `--yes`; human mode prompts without it. Feedback JSON combines aggregate and individual results: ```json theme={null} { "summary": { "average_rating": 4.5, "total_reviews": 2 }, "reviews": [] } ``` ## Traces and current session details `traces` uses the current REST session pipeline: * Summary list: `GET /api/v1/sessions` * Turn or span detail: `GET /api/v1/sessions/{session_id}` ```bash theme={null} observal ops traces --limit 20 --output json observal ops traces --platform kiro --days 7 --output json observal ops traces --turn --limit 5 --output json observal ops traces --span --limit 3 --output json ``` `--platform` accepts a registered harness. `--days` accepts 1 through 365 and `--limit` accepts 1 through 200. Default JSON returns the direct session summary array. Turn and span JSON fetch every selected session detail: ```json theme={null} { "view": "span", "items": [ { "summary": { "session_id": "session-id" }, "detail": { "events": [] } } ] } ``` `--turn` renders prompts and tool calls. `--span` includes full assistant and tool-result detail. Detail failures are surfaced rather than replaced with incomplete summaries. ## Telemetry status ```bash theme={null} observal ops telemetry status --output json ``` The server portion requires administrator access. The result combines recent server counts and local durable outbox state: ```json theme={null} { "server": { "status": "ok", "tool_call_events": 8, "agent_interaction_events": 5 }, "outbox": { "available": true, "pending": 0, "failed": 0, "total": 0, "bytes": 0 } } ``` A local outbox read failure remains visible as `available: false` with a safe error type. The removed synthetic test command is not part of the current JSONL session ingestion architecture. ## Logs Read a finite local tail: ```bash theme={null} observal ops logs --level WARNING --lines 50 --no-follow --output json ``` Follow local logs: ```bash theme={null} observal ops logs --level INFO --output json ``` Follow remote server logs with administrator access: ```bash theme={null} observal ops logs --remote --level WARNING --output json ``` Valid levels are `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`. `--filter` performs case-insensitive text filtering. `--lines` must be zero or greater. JSON mode emits JSON Lines. Local records have this shape: ```json theme={null} {"event":"log","source":"local","level":"ERROR","line":"raw log line"} ``` Remote records wrap the structured server event: ```json theme={null} {"event":"log","source":"remote","log":{"level":"ERROR","event":"request failed"}} ``` Malformed remote records, inaccessible files, HTTP errors, timeouts, and connection failures use categorized exits. Access tokens are never emitted. Log content itself may contain sensitive application data and must be handled accordingly. ## Agent insights List reports: ```bash theme={null} observal ops insights list alice/reviewer --output json ``` Show a report or one section: ```bash theme={null} observal ops insights show alice/reviewer latest --output json observal ops insights show alice/reviewer latest --section suggestions --output json ``` A section request returns `report_id`, `section`, and that section's `data` rather than the entire report. Generate once: ```bash theme={null} observal ops insights generate alice/reviewer --period 14 --output json ``` Generate and wait with JSON Lines progress: ```bash theme={null} observal ops insights generate alice/reviewer --period 14 --wait --output json ``` ```json theme={null} {"event":"queued","report":{"id":"report-id","status":"pending"}} {"event":"progress","report":{"id":"report-id","status":"running","progress_percent":50}} {"event":"progress","report":{"id":"report-id","status":"completed","progress_percent":100}} ``` Periods accept 1 through 365 days. `--version` and `--compare` require semantic versions. Invalid report rows, ambiguous prefixes, missing reports, unknown sections, unavailable providers, generation failures, and wait timeouts use categorized errors. Report sections include: * `at_a_glance` * `what_they_work_on` * `interaction_style` * `usage_patterns` * `what_works` * `friction_analysis` * `suggestions` * `usage_cost_analysis` * `version_comparison` * `regression_detection` * `on_the_horizon` * `fun_ending` ## Exit codes | Code | Meaning | | - | - | | 3 | Authentication required or failed | | 4 | Telemetry, log, session, or insight permission denied | | 5 | Registry item, report, session, or local log file not found | | 6 | Ambiguous report or Registry reference | | 7 | Invalid type, harness, section, period, version, comment, or missing confirmation | | 8 | Rate limit reached | | 9 | Server, provider, stream, outbox, or local log dependency unavailable | | 10 | CLI and server version mismatch | ## Related * [`observal inbox`](/cli/inbox): completed insight notifications * [`observal agent`](/cli/agent): apply insight recommendations * [`observal registry`](/cli/registry): inspect rated or recommended components * [Debug Agent failures](/use-cases/debug-agent-failures) # observal outdated Source: https://docs.observal.io/cli/outdated Compare versions recorded in `~/.observal/lockfile.json` with the latest versions available from the active Observal registry. The command never installs an update. ## Synopsis ```bash theme={null} observal outdated observal outdated --harness claude-code observal outdated --output json --no-report ``` ## Checked items The command checks: * Pulled agents * Separately installed MCP servers * Separately installed skills * Separately installed hooks Components bundled inside a pulled agent are not checked independently. Agent versions pin exact component versions, so update the agent when a newer agent version is reported. The exception is an agent version released before pinning: a component it never locked installs at its latest approved release, with a warning, and `--strict` refuses it. Agent authors use [`observal agent outdated`](/cli/agent#check-component-pins) to see which pinned components have newer releases and which are unlocked. A plain `observal agent pull` keeps an installed agent on its locked version, so the upgrade command for an agent includes `--upgrade`. The command reads only the active registry section of the lockfile. It requires authenticated registry access to retrieve current versions. ## Options | Option | Description | | - | - | | `--harness ` | Check one registered harness instead of every harness in the active lockfile section | | `--output table` | Render a Rich table; this is the default | | `--output json` | Emit one machine-readable JSON document | | `--report` | Send outdated findings to the user Inbox; this is the default | | `--no-report` | Suppress the Inbox write while still contacting the registry for version checks | Valid harnesses are `cursor`, `kiro`, `claude-code`, `codex`, `copilot`, `copilot-cli`, `opencode`, `antigravity`, `goose`, and `pi`. ## Table output Table output includes every installed item that was checked. Each row has one of these statuses: | Status | Meaning | | - | - | | `outdated` | A newer registry version exists | | `current` | The pinned version is current or newer | | `missing` | The pinned item no longer exists in the active registry | | `unknown` | The lockfile has no version for the item; reinstall it with the suggested command to record one | Outdated and unknown rows are followed by a type-specific command using the canonical `namespace/slug` identity. Agents use `observal agent pull --upgrade`; standalone components use their matching registry install command. A missing item is an item-level result, not a command failure, so a completed comparison containing missing rows exits successfully. ## JSON output JSON output has a stable top-level object: ```json theme={null} { "items": [ { "id": "11111111-1111-4111-8111-111111111111", "qualified_name": "acme/reviewer", "name": "reviewer", "namespace": "acme", "slug": "reviewer", "type": "agent", "harness": "claude-code", "current_version": "1.0.0", "latest_version": "2.0.0", "status": "outdated", "outdated": true, "error": null, "upgrade_command": "observal agent pull acme/reviewer --harness claude-code --no-prompt --upgrade" } ], "summary": { "total": 1, "outdated": 1, "current": 0, "missing": 0, "unknown": 0 }, "report": { "requested": true, "attempted": true, "succeeded": true, "created": 1, "superseded": 0, "error": null } } ``` An empty lockfile returns the same shape with an empty `items` array and zero summary counts. Missing rows include a categorized item-level error with any available request ID. Command-level JSON errors are written to stderr and leave stdout empty. ## Inbox reporting With reporting enabled, outdated findings are sent to the signed-in user's Inbox after the comparison. Duplicate findings are deduplicated by item and latest version. Inbox reporting is best-effort. If the comparison succeeds but reporting fails, the comparison still exits successfully and the `report` object contains the categorized reporting error. Human output prints a warning. Unexpected reporting failures are not suppressed. Use `--no-report` when no Inbox mutation is wanted. This does not make the command offline because registry reads are still required. ## Exit codes | Code | Meaning | | - | - | | `0` | Comparison completed, including empty, current, outdated, or missing results | | `1` | Unexpected or uncategorized failure, including an unexpected reporting failure | | `2` | Invalid command syntax or output mode | | `3` | Authentication is missing or invalid | | `4` | Registry or filesystem permission denied | | `7` | Invalid harness, malformed lockfile, or invalid installed version | | `8` | Registry rate limit reached | | `9` | Registry unavailable, timed out, or returned invalid version data | | `10` | CLI and server versions are incompatible | ## Related * [`observal agent pull`](/cli/pull): update a pulled agent * [`observal registry`](/cli/registry): install a standalone component * [`observal config`](/cli/config): inspect the active server configuration # observal registry prompt Source: https://docs.observal.io/cli/prompt Manage reusable prompt templates in the Registry. Prompts can be submitted, listed, rendered, edited, archived, restored, transferred, and shared with co-authors. Standalone prompt installation has been removed; attach prompts to agents instead. ## Commands | Command | Description | | - | - | | `submit` | Submit a prompt or save a draft | | `list` | List approved visible prompts | | `my` | List your prompts across all statuses | | `show` | Show one prompt, visible source provenance, and public fork count | | `fork` | Copy an approved prompt release into an independent draft | | `render` | Render a prompt with variables | | `edit` | Edit a draft, pending, or rejected prompt | | `archive` | Archive an approved prompt | | `unarchive` | Restore an archived prompt | | `transfer-owner` | Transfer ownership | | `co-authors` | List, add, or remove co-authors | All structured commands support table and JSON output. ## Submit ```bash theme={null} observal registry prompt submit \ --name review \ --description "Review code" \ --category code-review \ --template "Review {{code}}" \ --output json observal registry prompt submit --from-file prompt.json --output json observal registry prompt submit --submit acme/review --output json ``` JSON mode never prompts. A plain template file requires explicit name, description, and category options. A JSON file may contain the complete payload. Valid categories are `system-prompt`, `code-review`, `code-generation`, `testing`, `documentation`, `debugging`, and `general`. ## List and show ```bash theme={null} observal registry prompt list --category code-review --output json observal registry prompt my --output json observal registry prompt show acme/review --output json ``` Row numbers are scoped to the latest Prompt list. Empty Prompt lists clear previous Prompt row references. Human output escapes template text so bracketed content is rendered literally. ## Fork ```bash theme={null} observal registry prompt fork acme/review --name review-custom --output json observal registry prompt edit FORK_UUID --template 'Review {{code}} and tests' --output json observal registry prompt submit --submit FORK_UUID --output json ``` Only approved source listings and versions can be forked. A fork has its own namespace, review status, and template; editing it does not change the original. Public fork counts exclude drafts and private teamspace forks. A source that the viewer cannot access is shown only as **Source unavailable**. ## Render ```bash theme={null} observal registry prompt render acme/review --var code=main.py --output json ``` Every variable must use `key=value` syntax with a non-empty key. JSON returns the direct server result. Human output prints the rendered prompt as literal text. ## Edit ```bash theme={null} observal registry prompt edit acme/review --description "Updated" --output json observal registry prompt edit acme/review --from-file updates.json --output json ``` Invalid categories, versions, files, and edit-lock conflicts preserve their shared error category and stable exit code. ## Agent usage Prompts are agent components rather than standalone harness installations: ```bash theme={null} observal agent add prompt observal agent build ``` ## Related * [`observal registry`](/cli/registry): complete Registry reference * [`observal agent`](/cli/agent): compose and publish agents # observal agent pull Source: https://docs.observal.io/cli/pull Install a complete Agent into a harness. Pull resolves the Agent version to install, asks the server for harness-native config built from the exact component versions that Agent version pinned, merges generated files safely, installs bundled skills and hooks, runs required harness setup, and records exact installed state in `observal.lock` and the local lockfile. ## Synopsis ```bash theme={null} observal agent pull --harness [OPTIONS] ``` Agent references may be UUIDs, canonical `namespace/slug`, unambiguous bare names, aliases, or row numbers from the latest Agent list. ## Examples ```bash theme={null} observal agent pull alice/reviewer --harness kiro --no-prompt --output json observal agent pull alice/reviewer --harness claude-code --scope project --dry-run --no-prompt --output json observal agent pull alice/reviewer --harness pi --version 1.2.3 --no-prompt --output json observal agent pull alice/reviewer --harness cursor --no-prompt --upgrade observal agent pull alice/reviewer --harness cursor --no-prompt --strict ``` ## Options | Option | Description | | - | - | | `--harness`, `-i` | Required target: `cursor`, `kiro`, `claude-code`, `codex`, `copilot`, `copilot-cli`, `opencode`, `antigravity`, `goose`, or `pi` | | `--dir`, `-d` | Project directory used to resolve generated paths | | `--dry-run`, `-n` | Return planned files and setup commands without changing disk or installation metadata | | `--scope` | `project` or `user`, only for harnesses that support explicit scope | | `--model` | Model ID, or `harness=model`; repeatable | | `--tools` | Claude Code tool allowlist | | `--refresh-models` | Refresh the model catalog before an interactive model picker | | `--no-prompt`, `-y` | Disable environment, header, scope, and model prompts | | `--env`, `-e` | MCP environment value in `NAME=VALUE` form; repeatable | | `--header`, `-H` | MCP header in `NAME=VALUE` form; repeatable | | `--version`, `-V` | Install this exact Agent version and lock it | | `--upgrade` | Install the latest approved Agent version instead of the locked one, and lock it | | `--strict` / `--no-strict` | Refuse an install that does not match its lock; defaults to `OBSERVAL_STRICT` | | `--output`, `-o` | Table or JSON output | Unknown harnesses, unsupported scopes, malformed assignments, unused harness model overrides, unsupported model or tool options, invalid versions, and `--upgrade` combined with `--version` fail locally with validation exit code 7. A project-scope pull of an Agent with MCP servers also fails this way for Goose, which only reads a user-level MCP config: pull with `--scope user` instead. ## Pinned versions Pulls are pinned at two levels. **Components.** Every Agent version pins each of its MCP servers, skills, hooks, prompts, and sandboxes to one exact component version, identified by its version id and a content digest. Pull always installs those pinned versions. A component that ships a newer version never changes what an existing Agent version installs; the Agent's author releases a new Agent version to move it (see [`observal agent release --refresh-components`](/cli/agent#release-and-versions)). **The Agent version.** Pull chooses the Agent version in this order: 1. `--version X`: exactly that version. 2. `--upgrade`: the latest approved version. 3. `observal.lock` in `--dir`: the version the project locked. 4. `~/.observal/lockfile.json`: the version this machine already installed for this harness and directory. 5. Otherwise, a first install: the latest approved version. A plain pull therefore keeps an existing install on its version after newer versions are approved, and says when one is available. Only `--upgrade` or `--version` moves it. ### `observal.lock` Project-scope pulls write `observal.lock` to `--dir`. Commit it: teammates and CI pulling the same Agent in that project install the same Agent version, and so the same component versions, until someone runs `--upgrade` or `--version` and commits the change. ```json theme={null} { "lock_version": 1, "agents": { "alice/reviewer": { "id": "11111111-1111-1111-1111-111111111111", "version": "1.2.3", "lock_digest": "sha256:…", "components": [ {"type": "mcp", "qualified_name": "acme/github", "version": "1.4.2", "digest": "sha256:…"} ] } } } ``` Entries are keyed by `namespace/slug` and carry the Agent's registry `id`, so an Agent that was renamed or transferred keeps its locked version; the next pull rewrites the entry under the new name. User-scope installs are not tied to a project and neither read nor write `observal.lock`. `--dry-run` never writes it. A malformed or unsupported `observal.lock` fails with validation exit code 7 before anything is installed. If `observal.lock` or the local lockfile names an Agent version this server does not have, pull fails with not-found exit code 5 and says which lock pinned it; `--upgrade` or `--version` moves past it and rewrites the lock. ### Strict mode Without strict mode, pull installs and warns when: * a component of an Agent version released before pinning has no lock, so its latest version is installed; * a pinned component version no longer matches the digest recorded when the Agent version was locked; * a pinned component version is not approved; * the Agent version no longer matches the lock digest recorded in `observal.lock`. `--strict`, or `OBSERVAL_STRICT=1` for CI, turns each of these into a conflict (exit code 6) before any file is written. The flag wins over the environment variable, so `--no-strict` is the escape hatch in a strict pipeline. A server that predates component locks cannot check any of this, so a strict pull against it fails with version-mismatch exit code 10 instead of installing unchecked. JSON mode cannot prompt and requires `--no-prompt`. Missing required component values fail before config generation with `error.result.needs_input: true` and a list of names and component labels. ## Secrets Pull discovers required MCP environment variables and headers from the Agent's components. Interactive mode prompts for missing values. Non-interactive mode uses matching `--env` and `--header` assignments and stops before installation when any required value is missing. Optional values may remain unset. Values are sent only in the installation request and generated config. They are not included in JSON results, success messages, traces, or error details. Prefer environment expansion or secure shell input so secrets do not remain in shell history. ## File safety Generated relative paths are confined to `--dir`. Home paths are allowed only for an explicit user-scope installation supported by that harness. Absolute paths, parent traversal, and symlink escapes are rejected before installation tracking is updated. Pull behavior by file type: * JSON MCP and hook sections merge into existing objects. * YAML sections merge only when the existing top level and target section are mappings. * TOML managed tables are replaced idempotently while unrelated tables remain. * Generated text, prompt, Agent, and hook config files use atomic replacement. * Malformed or structurally incompatible existing config is never overwritten. The command exits with conflict code 6 and leaves it untouched. No `OTEL_*` or harness telemetry environment variables are generated. Session telemetry continues through Observal-managed hooks and reconciliation. ## Installation sequence Pull performs these steps: 1. Validate harness, scope, model, tool, assignment, version, and output combinations. 2. Resolve the canonical Agent, collect install options, and choose the Agent version (see [Pinned versions](#pinned-versions)). 3. Load MCP environment and header requirements from the pinned component versions. 4. Check installed component version conflicts. 5. Request the harness-specific installation config and its lock; refuse here in strict mode. 6. Resolve and validate every generated path. 7. Write or preview files and install bundled skills. 8. Run required harness MCP registration commands. 9. Record the installed Agent and component versions in the Registry-scoped lockfile and, for project-scope installs, in `observal.lock`. 10. Refresh the local layer snapshot and active-Agent state. Failed skill installation or MCP setup prevents installation metadata from being recorded. Setup commands have a 60-second timeout; a timeout is reported as a setup failure with exit code 9. A lockfile write failure is also reported as exit code 9 instead of claiming success. Failures after filesystem changes include safe partial state under `error.result`, including the stage, written file statuses, setup executable status, and tracking state. Setup arguments and secret values are omitted. A layer-snapshot failure is returned as a visible warning because the generated harness installation remains usable. ## JSON result Successful JSON output has this shape: ```json theme={null} { "agent": { "id": "11111111-1111-1111-1111-111111111111", "qualified_name": "alice/reviewer", "version": "1.2.3", "latest_version": "1.3.0", "resolved_from": "project-lock", "local_name": "reviewer" }, "project_lock": "/work/project/observal.lock", "lock": { "status": "locked", "digest": "sha256:…", "components": [ { "type": "mcp", "name": "GitHub", "id": "22222222-2222-2222-2222-222222222222", "version": "1.4.2", "version_id": "33333333-3333-3333-3333-333333333333", "digest": "sha256:…", "qualified_name": "acme/github", "source": "lock" } ], "problems": [] }, "harness": "kiro", "scope": "project", "dry_run": false, "target_directory": "/work/project", "files": [ { "path": "/work/project/.kiro/agents/reviewer.json", "status": "created" } ], "warnings": [], "setup_commands": [], "reports_sessions": true } ``` File statuses include `created`, `updated`, `merged`, `installed`, `cloned`, `would write`, and `would clone`. `agent.version` is the version that was installed and `agent.resolved_from` says why: `requested`, `upgrade`, `project-lock`, `installed`, or `latest`. `lock.status` is `locked`, `partial`, or `unlocked`; each component's `source` is `lock`, `version` (matched by its recorded version string), or `fallback-latest`. `lock.problems` lists what strict mode would refuse. `project_lock` is null for user-scope installs and dry runs. `reports_sessions` is true when written hook files contain Observal session push commands, including hooks retained during a merge. These hooks can report prompts, tool calls, and tool output to the configured server when they run, whether or not `observal doctor patch` was run. It does not guarantee successful delivery. For Pi, it is also true when the Observal Pi extension is installed, because the extension reports every Pi session. On a failed pull, check `error.result.reports_sessions` when available: a hook file may already be on disk and active even when this pull wrote no files and the Agent was not recorded as installed. Dry-run returns the same shape with `dry_run: true`, planned statuses, and `would_run` setup actions. In dry-run, `reports_sessions` predicts whether session hooks **would** be present after applying the plan; it does not mean the preview installed them. Dry-run does not write files, execute setup commands, update the lockfile or `observal.lock`, persist an active Agent, or emit a pull audit event. ## Human output Human mode lists every created, updated, merged, installed, cloned, or planned path. Component version conflicts, server warnings, snapshot warnings, and setup commands are printed explicitly. When session push hooks are configured, a telemetry line names the server they can report to. ## Exit codes | Code | Meaning | | - | - | | 3 | Authentication required or failed | | 4 | Agent or component access denied | | 5 | Agent or component not found | | 6 | Existing config cannot be merged safely, or a strict install does not match its lock | | 7 | Invalid harness, scope, version, path, assignment, or option combination | | 8 | Rate limit reached | | 9 | Server, filesystem, skill source, lockfile, or setup command unavailable | | 10 | CLI and server version mismatch, including a strict pull against a server without component locks | ## Related * [`observal agent`](/cli/agent): create and publish Agents * [`observal scan`](/cli/scan): inspect installed harness content * [`observal outdated`](/cli/outdated): compare installed Agent versions * [`observal agent outdated`](/cli/agent#check-component-pins): see which components an Agent version pins behind their latest release * [`observal doctor`](/cli/doctor): verify hooks and local installation state # observal registry recommend Source: https://docs.observal.io/cli/recommend List Registry components ranked for the signed-in user and record recommendation feedback. ## List The direct and explicit list forms are equivalent: ```bash theme={null} observal registry recommend --output json observal registry recommend list --limit 12 --type mcp --refresh --output json ``` | Option | Description | | - | - | | `--limit`, `-n` | Number of recommendations, from 1 through 24 | | `--type`, `-t` | Restrict to MCP, Skill, Hook, Prompt, or Sandbox | | `--refresh` | Recompute the work profile instead of using its cache | | `--output`, `-o` | Table or JSON output | JSON returns the direct server object. Important fields include: | Field | Meaning | | - | - | | `personalized` | Whether ranking used the user's sessions | | `profile_sessions` | Session count behind the profile | | `topics` | Dominant detected work topics | | `items` | Recommended visible components | | `items[].reason` | Server-provided explanation | | `items[].matched_on` | Terms from the user's own work profile | An empty `items` array is a successful result. It means either no visible components remain or everything applicable has already been installed or dismissed. When `personalized` is false, results are popularity-based and must not be described as tailored. ## Dismiss ```bash theme={null} observal registry recommend dismiss skill acme/reviewer --output json observal registry recommend dismiss mcp acme/postgres --action not_relevant --output json observal registry recommend dismiss hook acme/guard --action installed --output json ``` Valid actions are `dismissed`, `not_relevant`, and `installed`. JSON returns: ```json theme={null} { "component_type": "mcp", "component_id": "498c17ac-1234-4567-89ab-cdef01234567", "action": "installed" } ``` Dismissals are scoped to the signed-in user. The CLI cannot inspect another user's recommendation profile. ## Exit codes Invalid component types and actions use validation exit code 7. Authentication, permission, rate-limit, unavailable-service, and version-mismatch failures preserve the shared CLI exit contract. JSON failures leave stdout empty and write one error object to stderr. ## Related * [`observal registry`](/cli/registry): browse components directly * [`observal outdated`](/cli/outdated): check installed versions # observal reconcile Source: https://docs.observal.io/cli/reconcile Backfill local session records missed by automatic hook or extension delivery. Reconcile is a manual recovery command, not the normal collection path. The server cannot scan files on a developer machine. Automatic hooks and extensions wake the same durable delivery engine during normal harness activity; reconcile scans recent local sources on demand. ## When to use it Run reconcile when: * Telemetry instrumentation was installed after sessions already existed * The machine or server was offline * Delivery was interrupted * Durable outbox records remain pending * Recent local history needs to be verified or backfilled Routine use is unnecessary when automatic delivery is healthy. ## Examples Preview the default seven-day window without network or cursor changes: ```bash theme={null} observal reconcile --dry-run --output json ``` Backfill every installed harness: ```bash theme={null} observal reconcile --output json ``` Target one harness and a 24-hour discovery window: ```bash theme={null} observal reconcile --harness kiro --since 24 --output json ``` The discovery window accepts 1 through 8,760 hours. ## Delivery behavior A non-dry run: 1. Validates configuration and the harness selection before any outbox side effect. 2. Retries the existing durable outbox. 3. Discovers recent session sources through installed harness adapters. 4. Skips locally finalized sources that have not grown. 5. Recovers the contiguous server checkpoint for unfinished sources. 6. Queues and sends only complete records after that checkpoint. 7. Sends final metadata when all records were uploaded but the session was not finalized. 8. Leaves transient failures queued for the next automatic wake-up or reconcile run. 9. Quarantines permanent server rejections and reports them explicitly. Acknowledged checkpoints make repeated runs idempotent. A dry run only reads local sources and cursor state. It does not drain the outbox, contact the ingest API, or update cursor state. ## JSON result ```json theme={null} { "dry_run": false, "since_hours": 168, "outbox_drained": true, "targets": [ { "harness": "kiro", "discovered": 2, "pushed": 1, "finalized": 0, "queued": 1, "rejected": 0, "would_push": 0, "would_finalize": 0, "up_to_date": 0, "skipped": 0, "errors": 0, "sessions": [ {"session_id": "session-1", "status": "pushed", "bytes_new": 512}, {"session_id": "session-2", "status": "queued", "bytes_new": 128} ] } ], "summary": { "discovered": 2, "pushed": 1, "finalized": 0, "queued": 1, "rejected": 0, "would_push": 0, "would_finalize": 0, "up_to_date": 0, "skipped": 0, "errors": 0 }, "rejections": [] } ``` `outbox_drained: false` means durable records remain pending. A queued session is safely stored for retry and is not reported as delivered. Permanent ingestion failures include their HTTP status in `rejections`. ## Exit codes | Code | Meaning | | - | - | | 3 | Session delivery identity is not configured | | 7 | Unknown harness or invalid discovery window | | 9 | Outbox storage or session discovery is unavailable | Per-session source read failures and checkpoint mismatches are explicit result items so other sessions can still be recovered. ## Related * [`observal doctor`](/cli/doctor): configure and verify automatic telemetry instrumentation * [`observal ops telemetry status`](/cli/ops): inspect server and durable outbox health * [Session tracking](/core-concepts/session-tracking): automatic and recovery delivery architecture # observal registry Source: https://docs.observal.io/cli/registry Publish and manage registry components. The registry has five component types: MCP servers, skills, hooks, prompts, and sandboxes. ## Subcommand structure ```text theme={null} observal registry [args] ``` | Type | Submit | List | My | Show | Install | Render | Edit | Fork | | - | - | - | - | - | - | - | - | - | | `mcp` | yes | yes | yes | yes | yes | no | yes | yes | | `skill` | yes | yes | yes | yes | yes | no | yes | yes | | `hook` | yes | yes | no | yes | yes | no | yes | yes | | `prompt` | yes | yes | yes | yes | no | yes | yes | yes | | `sandbox` | yes | yes | no | yes | no | no | yes | yes | Every component type also supports archive, unarchive, ownership transfer, and co-author management. Registry also contains the `models`, `version`, `recommend`, and mixed `bulk` groups. All registry references accept a UUID, canonical `namespace/slug`, a unique legacy bare name, a row number from the latest human list output for the same component type, or an `@alias`. Agents and scripts must use returned UUIDs or `qualified_name` values, never row numbers. If the same bare slug exists in multiple namespaces, qualify it, for example `alice/search` instead of `search`. ### Fork an approved component ```bash theme={null} observal registry skill fork-listing acme/review-skill --name my-review-skill --output json observal registry mcp fork acme/github --version 1.2.0 --new-version 0.1.0 --output json observal registry hook fork acme/audit --team platform --visibility team --output json observal registry prompt fork acme/review --output json observal registry sandbox fork acme/python --output json ``` The five `fork` commands accept `SOURCE [--name NAME] [--version BASE] [--new-version V] [--team HANDLE|UUID] [--visibility public|team] [--output table|json]`. Only currently approved sources and approved base versions are eligible. The fork is a separate editable **draft**, not a new release of the original; it has its own namespace, owner, reviews, and installation identity. Team-private sources can only be forked within their teamspace. JSON returns the direct server draft object. Edit using `observal registry edit FORK_UUID ...` and submit through that type's draft workflow. `show` prints visible fork provenance and the count of public approved direct forks. If source access is lost, it displays **Source unavailable** rather than the saved source reference. A private or unpublished fork never increases the public count. ### Shared lifecycle and collaboration commands ```bash theme={null} observal registry skill archive alice/reviewer --yes --output json observal registry skill unarchive alice/reviewer --yes --output json observal registry skill transfer-owner alice/reviewer bob --yes --output json observal registry skill co-authors list alice/reviewer --output json observal registry skill co-authors add alice/reviewer bob@example.com --output json observal registry skill co-authors remove alice/reviewer --output json ``` Archive, restore, and ownership transfer require explicit confirmation in JSON mode. Their JSON output is the direct server result. Co-author list returns the standard list envelope; add and remove return the direct server result. The namespace is the publisher's username or a teamspace handle. Usernames cannot change after the account owns a registry listing. Team members can browse approved private teamspace items in normal list results. Use `--team TEAM_HANDLE` to include public items plus that team's private items, or `--namespace TEAM_HANDLE` to restrict results to that namespace. Direct references use `team-handle/item-slug`. Nonmembers receive the same not-found response for private items as for unknown items. ### Teamspace visibility Use the teamspace target and visibility options on submit commands: ```bash theme={null} observal registry skill submit --skill-md ./SKILL.md --team platform-tools --visibility public observal registry skill submit --skill-md ./SKILL.md --team platform-tools --visibility team observal registry skill list --team platform-tools --search 'frontend design' --harness claude-code --output json observal registry skill list --namespace platform-tools --output json observal registry skill show platform-tools/internal-skill --output json ``` `public` teamspace items are visible to all registry users. `team` items are visible only to team members and privileged reviewers. Team owners and reviewers can change visibility after publication. A team member's new submission still follows the normal review workflow. *** ## Mixed bulk submission Submit up to 200 MCP, skill, hook, prompt, and sandbox entries from one JSON file: ```bash theme={null} observal registry bulk submit --from-file components.json --dry-run --output json observal registry bulk submit --from-file components.json --yes --output json ``` The file is a bare array or an object with a `components` array. Each entry contains `type` plus the normal API submission fields: ```json theme={null} { "components": [ { "type": "skill", "name": "review-helper", "version": "1.0.0", "description": "Reviews changes", "owner": "alice", "task_type": "code-review", "delivery_mode": "registry_direct", "skill_md_content": "---\nname: review-helper\ndescription: Reviews changes\n---\n" }, { "type": "prompt", "name": "review-brief", "description": "Review prompt", "owner": "alice", "category": "general", "template": "Review {{change}}" } ] } ``` Dry run validates file structure without contacting submission endpoints. Execution structurally validates every entry before the first mutation, submits entries in order, reports conflicts as skipped, and returns per-entry IDs, canonical names, review status, and safe errors. Results include `partial`; item errors set it to `true` and exit with code 11, while conflicts counted as skips remain successful. Authentication, permission, rate-limit, version, and service failures stop the batch. JSON execution requires `--yes`. Re-running a partially completed file is safe only after inspecting results. Existing identities are skipped for component types that reject duplicates. Verify created items by returned UUID or `qualified_name`. *** ## MCP servers MCP server registry commands for submitting, browsing, generating configuration, editing, and archiving MCP server listings. ### `observal registry mcp submit` Submit an MCP server to the registry. By default, paste your server's JSON config (the same format you use in your harness). Use `--git` to analyze a git repository instead. #### Synopsis ```bash theme={null} observal registry mcp submit [OPTIONS] observal registry mcp submit --git [OPTIONS] ``` #### Options | Option | Short | Description | | - | - | - | | `--git` | `-g` | Analyze a git repository instead of pasting config | | `--name` | `-n` | Pre-fill server name (skip prompt) | | `--category` | `-c` | Pre-fill category (skip prompt) | | `--yes` | `-y` | Accept all defaults | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit an existing draft for review (MCP ID) | | `--output` | `-o` | Output format: `table` or `json` | #### Default flow (JSON paste) 1. Prompts you to paste your MCP server JSON config. 2. Accepts multiple formats: * **harness config**: `{"mcpServers": {"name": {"command": "...", "args": [...], "env": {...}}}}` * **Bare config**: `{"command": "npx", "args": ["-y", "pkg"]}` * **SSE/HTTP**: `{"url": "http://...", "type": "sse", "headers": {...}}` * **server.json manifest**: `{"packages": [...], "remotes": [...]}` 3. Auto-detects environment variables from `$VAR` patterns and `env` keys. 4. Shows a config preview and prompts for metadata (name, description, category). 5. Submits to registry for review. #### Git analysis flow (`--git`) 1. Shallow-clones the repo. 2. Detects the MCP framework (FastMCP, MCP SDK, TypeScript SDK, Go SDK). 3. Extracts server name, description, and exposed tools via AST. 4. Scans for required env vars (`os.environ`, `os.getenv`, `.env.example`, `server.json`). 5. Prompts for metadata confirmation. 6. Submits to registry for review. #### Examples ```bash theme={null} # Paste config (default, recommended) observal registry mcp submit # Non-interactive with piped JSON echo '{"command": "npx", "args": ["-y", "@example/mcp-server"]}' | observal registry mcp submit -y -n my-server -c developer-tools --output json # Save as draft observal registry mcp submit --draft # Analyze a git repo observal registry mcp submit --git https://github.com/MarkusPfundstein/mcp-obsidian # Non-interactive git analysis observal registry mcp submit --git https://github.com/sooperset/mcp-atlassian -y # Submit an existing draft for review observal registry mcp submit --submit my-server ``` #### Valid categories `browser-automation`, `cloud-platforms`, `code-execution`, `communication`, `databases`, `developer-tools`, `devops`, `file-systems`, `finance`, `knowledge-memory`, `monitoring`, `multimedia`, `productivity`, `search`, `security`, `version-control`, `ai-ml`, `data-analytics`, `general`. #### Valid transports `stdio`, `sse`, `streamable-http`. The older `http` spelling is accepted and stored as `streamable-http`; any other value is rejected. #### Valid frameworks `python`, `docker`, `typescript`, `go`. *** ### `observal registry mcp list` List approved MCP servers in the registry. ```bash theme={null} observal registry mcp list [--search TERM] [--category CAT] [--limit N] [--sort name|category|version] [--output table|json] [--interactive] ``` | Option | Short | Description | | - | - | - | | `--search` | `-s` | Search by name or description | | `--category` | `-c` | Filter by category | | `--limit` | `-n` | Max results (default: 50) | | `--sort` | | Sort by: `name`, `category`, `version` | | `--output` | `-o` | Output format: `table`, `json` | | `--interactive` | `-i` | Open a fuzzy-search picker | ```bash theme={null} observal registry mcp list --search github observal registry mcp list --category ai-ml --output json observal registry mcp list --interactive observal registry mcp list --sort category --limit 10 ``` *** ### `observal registry mcp my` List your own MCP servers across all statuses (draft, pending, approved, rejected). ```bash theme={null} observal registry mcp my [--output table|json] ``` ```bash theme={null} observal registry mcp my observal registry mcp my --output json ``` *** ### `observal registry mcp show` Show full details of an MCP server including validation results, env vars, and supported harnesses. ```bash theme={null} observal registry mcp show [--output table|json] ``` ```bash theme={null} observal registry mcp show my-server observal registry mcp show 3 observal registry mcp show @fav --output json ``` *** ### `observal registry mcp install` Generate a harness config snippet for an MCP server. This command does not write harness configuration or record an installation. Interactive mode prompts for required environment variables and headers. JSON mode requires `--no-prompt` and rejects missing required values before config generation with `error.result.needs_input: true`. ```bash theme={null} observal registry mcp install --harness [options] ``` | Option | Short | Description | | - | - | - | | `--harness` | `-i` | Target harness (required) | | `--version` | `-V` | Generate configuration from that exact version instead of the latest | | `--env` | `-e` | Environment value as `KEY=VALUE`; repeatable | | `--header` | | Header value as `KEY=VALUE`; repeatable | | `--env-file` | | Read environment values from a file | | `--no-prompt` | `-y` | Use supplied values without prompting; JSON fails if required values are missing | | `--raw` | | Output only a bare config template; missing values remain explicit placeholders | | `--output` | `-o` | Output the complete operation result as table or JSON | ```bash theme={null} observal registry mcp install my-server --harness claude-code observal registry mcp install my-server --harness cursor --raw > .cursor/mcp.json observal registry mcp install 2 --harness copilot observal registry mcp install @db --harness kiro ``` *** ### `observal registry mcp edit` Edit an MCP server submission. For draft/pending/rejected listings, edits in place. For approved listings, publishes a new version with a semver bump. ```bash theme={null} observal registry mcp edit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Load updates from a JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--category` | `-c` | New category | | `--version` | `-v` | New version string | | `--git-url` | | New git URL | | `--command` | | New command | | `--url` | | New URL (SSE/HTTP) | | `--bump` | | Version bump for approved listings: `patch`, `minor`, or `major` | | `--changelog` | | Changelog for an approved-listing version | | `--output` | `-o` | Output format: `table` or `json` | Without flags, opens an interactive JSON paste prompt (same format as submit). ```bash theme={null} # Interactive JSON paste edit observal registry mcp edit my-server # Update specific fields observal registry mcp edit my-server -d "New description" -c databases # Load updates from a file observal registry mcp edit my-server --from-file updates.json # Bump version on an approved listing observal registry mcp edit my-server --version 1.2.0 ``` *** ### `observal registry mcp transfer-owner` Transfer ownership to another username. You stop being the owner immediately. ```bash theme={null} observal registry mcp transfer-owner my-server @alice -y ``` *** ### `observal registry mcp sync` Publish a new MCP version automatically when the listing's GitHub or GitLab repository changes. See [Sync an MCP server from GitHub or GitLab](/use-cases/mcp-repo-sync) for the full setup. | Command | Description | | - | - | | `enable ` | Turn on sync, or change its triggers. Prints the webhook URL, and the secret on first enable | | `status ` | Show the triggers, tracked branch, and last sync result | | `run ` | Sync the tracked branch now | | `rotate-secret ` | Issue a new webhook secret | | `disable ` | Turn off sync. Published versions are kept | | `enable` option | Short | Description | | - | - | - | | `--push / --no-push` | | Publish a version on each push to the tracked branch (on by default) | | `--release / --no-release` | | Publish a version when a release is published (off by default) | | `--branch` | `-b` | Branch to track. Defaults to the repository's default branch | | `--provider` | | `github` or `gitlab`. Detected from the repository URL; set it for a self-managed GitLab whose hostname does not contain `gitlab` | | `--output` | `-o` | Output format: `table` or `json` | On an existing setup, `enable` keeps any setting you do not pass. ```bash theme={null} # Push sync on the default branch observal registry mcp sync enable alice/weather-mcp # Releases only observal registry mcp sync enable alice/weather-mcp --release --no-push # Self-managed GitLab on its own hostname observal registry mcp sync enable alice/weather-mcp --provider gitlab # Check the last result observal registry mcp sync status alice/weather-mcp --output json ``` *** ## Skills Skill registry commands. Skills are portable SKILL.md instruction packages that provide agents with task-specific guidance. Valid task types: `code-review`, `code-generation`, `testing`, `documentation`, `debugging`, `refactoring`, `deployment`, `security-audit`, `performance`, `general`. ### `observal registry skill submit` Submit a new skill for review. Provide `--git-url` to let the server fetch SKILL.md automatically, or use `--skill-md` to paste content directly. ```bash theme={null} observal registry skill submit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Create from JSON file | | `--skill-md` | | Path to SKILL.md (auto-fills fields from frontmatter) | | `--git-url` | | Git repository URL | | `--git-ref` | | Branch or tag (default: main) | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (skill ID) | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry skill submit --git-url https://github.com/org/repo observal registry skill submit --from-file skill.json observal registry skill submit --skill-md ./SKILL.md --git-url https://github.com/org/repo --name review --description "Review code" --task-type code-review --output json observal registry skill submit --draft observal registry skill submit --submit abc123 ``` *** ### `observal registry skill list` List approved skills in the registry. ```bash theme={null} observal registry skill list [--task-type TYPE] [--target-agent AGENT] [--search TERM] [--output table|json] ``` | Option | Short | Description | | - | - | - | | `--task-type` | `-t` | Filter by task type | | `--target-agent` | | Filter by target agent | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash theme={null} observal registry skill list observal registry skill list --task-type code-review observal registry skill list --target-agent claude-code --output json observal registry skill list --search "refactor" ``` *** ### `observal registry skill my` List your own skills across all statuses (draft, pending, approved, rejected). ```bash theme={null} observal registry skill my [--output table|json] ``` ```bash theme={null} observal registry skill my observal registry skill my --output json ``` *** ### `observal registry skill show` Show detailed information about a skill, including validation status, task type, git source, and slash command. ```bash theme={null} observal registry skill show [--output table|json] ``` ```bash theme={null} observal registry skill show my-skill observal registry skill show 1 observal registry skill show @refactor-skill --output json ``` *** ### `observal registry skill install` Install a skill by fetching the full skill directory from git. Clones the skill directory via sparse checkout and writes it to the appropriate harness skill path. ```bash theme={null} observal registry skill install --harness [--scope user|project] [--raw] [--no-write] ``` | Option | Short | Description | | - | - | - | | `--harness` | `-i` | Target harness (required) | | `--scope` | `-s` | Install scope: `user` (global, default) or `project` | | `--raw` | | Output raw JSON only | | `--no-write` | | Generate config without writing skill files or lockfile state | | `--version` | `-V` | Install one version instead of the latest | | `--output` | `-o` | Output the operation result as table or JSON | Scopes: * `user` (default): writes to `~/./skills//` globally. * `project`: writes to `.agents/skills//` in the current directory, then symlinks into detected harness config directories. JSON output does not disable installation. It returns whether files were written and the installed path. Raw and no-write modes do not record the skill as installed. An installed skill is recorded with its exact version, version id, and digest, with or without `--version`. A failed file write or lockfile update returns a categorized failure instead of reporting success. ```bash theme={null} observal registry skill install my-skill --harness claude-code observal registry skill install @sk --harness kiro --scope project observal registry skill install 2 --harness cursor --raw observal registry skill install my-skill --harness antigravity --no-write ``` *** ### `observal registry skill edit` Edit a draft, pending, or rejected skill submission. Acquires an edit lock to prevent concurrent modifications. ```bash theme={null} observal registry skill edit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--task-type` | `-t` | New task type | | `--git-url` | | New git URL | | `--git-ref` | | New git ref | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry skill edit my-skill --description "Better desc" observal registry skill edit abc123 --from-file updates.json observal registry skill edit @sk --git-url https://github.com/org/new-repo observal registry skill edit 2 --version 2.0.0 --task-type debugging ``` *** ### `observal registry hook submit` Submit a new hook for review. Supports inline script content via `--script`, or git-hosted hooks via `--source-url`. ```bash theme={null} observal registry hook submit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Create from JSON file | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (hook ID) | | `--script` | | Path to hook script file (content stored in registry) | | `--source-url` | | Git repo containing hook scripts | | `--source-ref` | | Branch/tag to track (default: main) | | `--source-path` | | Directory within repo containing hook files | | `--requires` | | Install prerequisites (repeatable) | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry hook submit observal registry hook submit --script ./protect-files.sh observal registry hook submit --source-url https://github.com/org/hooks --source-path hooks/guard/ observal registry hook submit --from-file hook.json observal registry hook submit --draft observal registry hook submit --submit abc123 ``` *** ### `observal registry hook list` List approved hooks from the registry. ```bash theme={null} observal registry hook list [--event EVENT] [--search TERM] [--output table|json] ``` | Option | Short | Description | | - | - | - | | `--event` | `-e` | Filter by event type | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash theme={null} observal registry hook list observal registry hook list --event Stop observal registry hook list --search guard --output json ``` *** ### `observal registry hook show` Show detailed information for a single hook, including event type, handler config, and execution mode. ```bash theme={null} observal registry hook show [--output table|json] ``` ```bash theme={null} observal registry hook show my-hook observal registry hook show 1 observal registry hook show @guard --output json ``` *** ### `observal registry hook install` Install a hook for a specific harness. Writes script files and merges hook config into the harness's settings. Existing hooks are preserved during merge. ```bash theme={null} observal registry hook install --harness [--platform PLATFORM] [--raw] [--dir DIR] [--version VERSION] ``` | Option | Short | Description | | - | - | - | | `--harness` | `-i` | Target harness (required) | | `--platform` | `-p` | Platform: `win32`, `darwin`, `linux` | | `--raw` | | Output raw JSON only (no file writes) | | `--dir` | `-d` | Project directory for file writes (default: cwd) | | `--version` | `-V` | Install one version instead of the latest | | `--output` | `-o` | Output the complete installation result as table or JSON | ```bash theme={null} observal registry hook install my-hook --harness claude-code observal registry hook install @guard --harness kiro --dir ./project observal registry hook install my-hook --harness cursor --raw observal registry hook install my-hook --harness claude-code --platform darwin observal registry hook install my-hook --harness claude-code --output json ``` Hook installation validates every path before writing, refuses to replace malformed existing JSON, writes files atomically, and does not duplicate an existing event entry when repeated. The lockfile records the installed version, its version id and digest, and any version you requested. *** ### `observal registry hook edit` Edit a draft, pending, or rejected hook submission. Acquires an edit lock to prevent concurrent modifications. ```bash theme={null} observal registry hook edit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--event` | `-e` | New event type | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry hook edit my-hook --description "Updated guard hook" observal registry hook edit my-hook --event Stop --version 1.1.0 observal registry hook edit @guard --from-file updated-hook.json observal registry hook edit 1 --name new-name ``` *** ### `observal registry prompt submit` Submit a new prompt template for review. You can submit interactively, from a JSON file, or from a raw template file. ```bash theme={null} observal registry prompt submit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Create from JSON file, or read template from a text file | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (prompt ID) | | `--output` | `-o` | Output format: `table` or `json` | If `--from-file` points to a non-JSON file, its content is used as the template and you are prompted for metadata interactively. ```bash theme={null} observal registry prompt submit observal registry prompt submit --from-file prompt.json observal registry prompt submit --from-file template.md observal registry prompt submit --draft observal registry prompt submit --submit abc123 ``` *** ### `observal registry prompt list` List approved prompts in the registry. ```bash theme={null} observal registry prompt list [--category CAT] [--search TERM] [--output table|json] ``` | Option | Short | Description | | - | - | - | | `--category` | `-c` | Filter by category | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash theme={null} observal registry prompt list observal registry prompt list --category code-review observal registry prompt list --search "refactor" --output json ``` *** ### `observal registry prompt my` List your own prompts across all statuses (draft, pending, approved, rejected). ```bash theme={null} observal registry prompt my [--output table|json] ``` ```bash theme={null} observal registry prompt my observal registry prompt my --output json ``` *** ### `observal registry prompt show` Show detailed information about a prompt, including the template content. ```bash theme={null} observal registry prompt show [--output table|json] ``` ```bash theme={null} observal registry prompt show my-prompt observal registry prompt show 1 observal registry prompt show @refactor-prompt --output json ``` *** ### `observal registry prompt render` Render a prompt template with variable substitution. Sends key=value pairs to the server, which substitutes them into the template and returns the rendered output. ```bash theme={null} observal registry prompt render --var key=value [--var key2=value2 ...] ``` | Option | Short | Description | | - | - | - | | `--var` | `-v` | Variable as `key=value` (repeatable) | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry prompt render my-prompt --var lang=python observal registry prompt render @tpl --var file=main.py --var task=refactor ``` *** ### `observal registry prompt edit` Edit a draft, pending, or rejected prompt submission. Acquires an edit lock to prevent concurrent modifications. ```bash theme={null} observal registry prompt edit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--category` | `-c` | New category | | `--template` | `-t` | New template text | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry prompt edit my-prompt --description "Updated desc" observal registry prompt edit abc123 --from-file updates.json observal registry prompt edit @tpl --template "New template: {{ var }}" observal registry prompt edit 2 --version 2.0.0 --category debugging ``` *** ### `observal registry sandbox submit` Submit a new sandbox environment for review. ```bash theme={null} observal registry sandbox submit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Create from JSON file | | `--draft` | | Save as draft instead of submitting for review | | `--submit` | | Submit a draft for review (sandbox ID) | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry sandbox submit observal registry sandbox submit --from-file sandbox.json observal registry sandbox submit --draft observal registry sandbox submit --submit abc123 ``` *** ### `observal registry sandbox list` List approved sandboxes in the registry. ```bash theme={null} observal registry sandbox list [--runtime TYPE] [--search TERM] [--output table|json] ``` | Option | Short | Description | | - | - | - | | `--runtime` | `-r` | Filter by runtime type | | `--search` | `-s` | Search by name or description | | `--output` | `-o` | Output format: `table`, `json` | ```bash theme={null} observal registry sandbox list observal registry sandbox list --runtime docker observal registry sandbox list --search "node" --output json ``` *** ### `observal registry sandbox show` Show detailed information about a sandbox, including runtime type, container image, and resource limits. ```bash theme={null} observal registry sandbox show [--output table|json] ``` ```bash theme={null} observal registry sandbox show my-sandbox observal registry sandbox show 1 observal registry sandbox show @dev-env --output json ``` *** Sandboxes are attached to agents by UUID and are installed when the agent is pulled. There is no standalone Sandbox install command. ```bash theme={null} observal agent add sandbox observal agent build ``` *** ### `observal registry sandbox edit` Edit a draft, pending, or rejected sandbox submission. Acquires an edit lock to prevent concurrent modifications. ```bash theme={null} observal registry sandbox edit [OPTIONS] ``` | Option | Short | Description | | - | - | - | | `--from-file` | `-f` | Load updates from JSON file | | `--name` | `-n` | New listing name | | `--description` | `-d` | New description | | `--version` | `-v` | New version string | | `--runtime-type` | `-r` | New runtime type | | `--image` | `-i` | New container image | | `--output` | `-o` | Output format: `table` or `json` | ```bash theme={null} observal registry sandbox edit my-sandbox --image node:20-alpine observal registry sandbox edit abc123 --from-file updates.json observal registry sandbox edit @env --runtime-type docker --version 2.0.0 --output json ``` *** ## Component versions Use `observal registry version publish` and `observal registry version list` for all five component types. Publication supports direct JSON results; history supports explicit pagination. See [`observal registry version`](/cli/component) for the complete contract. ## Personalized recommendations Use `observal registry recommend` to rank visible components against the signed-in user's sessions and to dismiss or mark recommendations as installed. See [`observal registry recommend`](/cli/recommend) for the JSON schema and feedback actions. # observal scan Source: https://docs.observal.io/cli/scan Discover MCP servers, hooks, and telemetry configuration across your harness configs. `scan` is **read-only** -- it shows what you have without modifying any files. To install session telemetry hooks, use [`observal doctor patch`](/cli/doctor). MCP commands and URLs are never rewritten. ## Synopsis ```bash theme={null} observal scan [--harness ] [--inventory] [--output table|json] ``` Use `--inventory` for **local-only**, bounded harness evidence. It returns a versioned JSON document with `inventory` and `diagnostics`, or a table showing the harness, scope, component type, name, and privacy-safe source path. URLs have userinfo, fragments, and **all query parameters** removed; suspicious credentials in the URL path make a launch incomplete. Token values, raw prompts, and command arguments are not printed. Unrecognized or unsafe launches have `launch: null` and a diagnostic, not a guessed Registry identity. An empty inventory succeeds with an empty list. It never contacts Observal, writes a lockfile, or submits a component. Package-manager installations alone are not scanned or treated as configured capabilities. Component names and source filenames are user-supplied, so review the output before sharing it externally. To find approved resources for a task, use [`observal discover`](/cli/discover). To submit something you found locally, **you** choose to run the corresponding `observal registry submit --draft` command; inventory never starts that workflow. ## Options | Option | Description | | - | - | | `--harness ` | Scope to one harness: `cursor`, `kiro`, `claude-code`, `codex`, `copilot`, `copilot-cli`, `opencode`, `antigravity`, `goose`, `pi` | | `--inventory` | Bounded, redacted **local-only** evidence; no Registry lookup. | | `--output table\|json` | Render a table or machine-readable JSON. | If you run `observal scan` with no flags, it auto-detects every installed harness and scans each in turn. ## What it does 1. Finds MCP config files: * Claude Code: `~/.claude/settings.json` * Kiro: `.kiro/settings/mcp.json` (project) or `~/.kiro/settings/mcp.json` (home) * Cursor: `.cursor/mcp.json` * Copilot: `.vscode/mcp.json` * Antigravity: `.agents/mcp_config.json` or `~/.gemini/antigravity-cli/mcp_config.json` * Goose: `~/.config/goose/config.yaml` (the `extensions` key) * Copilot CLI: `.mcp.json` (project) or `~/.copilot/mcp-config.json` (home) 2. Lists every MCP server found and its direct command or URL. 3. Reports installed session telemetry hooks. No files are written and no registration happens. **Default table-mode scan may query the Registry when authenticated** to show unregistered names; use `--inventory` for a guaranteed local-only scan. ## Example ```bash theme={null} observal scan ``` Output: ``` Claude Code (~/.claude/settings.json) filesystem npx @modelcontextprotocol/server-filesystem not wrapped github npx @modelcontextprotocol/server-github not wrapped Kiro (.kiro/settings/mcp.json) mcp-obsidian mcp-obsidian not wrapped 2 harness(s) found, 3 MCP server(s) total, 0 wrapped. ``` ## Scoping to a single harness ```bash theme={null} observal scan --harness claude-code ``` ## What to do next Once you see what's installed, instrument it: ```bash theme={null} # Install session telemetry hooks across all harnesses observal doctor patch --all-harnesses # Or target a specific harness observal doctor patch --harness kiro # Preview changes without writing anything observal doctor patch --all-harnesses --dry-run ``` ## Exit codes | Code | Meaning | | - | - | | 0 | Inventory completed (even when empty); default scan found a harness config | | 1 | Unknown harness; default scan found no harness configs | ## Related * [`observal doctor patch`](/cli/doctor): instrument your harnesses (hooks, shims) * [`observal agent pull`](/cli/pull): install a full agent (also wires up MCP servers) * [`observal doctor`](/cli/doctor): diagnose instrumentation end-to-end * [Use Cases -- Observe MCP traffic](/use-cases/observe-mcp-traffic): narrative walkthrough # observal self Source: https://docs.observal.io/cli/self Inspect or change the installed CLI version. All four workflows support `--output table|json`. JSON upgrade, downgrade, and rollback require `--force` and never prompt. ## Commands | Command | Purpose | | - | - | | `upgrade` | Install the latest or a specified newer CLI release | | `downgrade` | List releases or install an older release | | `rollback` | Restore the standalone binary saved before the last version change | | `status` | Show current version, install method, and update availability | The destructive uninstall workflow has been removed. ## Status ```bash theme={null} observal self status --output json ``` JSON returns: ```json theme={null} { "current_version": "2.4.0", "install_method": "uv_tool", "path": "/home/user/.local/bin/observal", "writable": true, "managed_by": "uv", "github_available": true, "latest_version": "2.5.0", "update_available": true } ``` The command always checks GitHub. An unreachable GitHub service remains a successful local status result with `github_available: false`, `latest_version: null`, and `update_available: null`. ## Upgrade Install the latest stable release: ```bash theme={null} observal self upgrade --force --output json ``` Install a specific release: ```bash theme={null} observal self upgrade --version 2.5.0 --force --output json ``` Include prereleases when resolving the latest version: ```bash theme={null} observal self upgrade --pre --force --output json ``` A completed JSON result contains the prior version, target version, install method, and executable path. When the current version already matches the target, status is `up_to_date` and no installation occurs. The command rejects invalid versions and targets older than the current version. System-package installations must be upgraded through their package manager. ## Downgrade List available releases: ```bash theme={null} observal self downgrade --list --output json ``` JSON returns `current_version` and release `items`, each with a `current` boolean. Install an older release: ```bash theme={null} observal self downgrade --version 2.4.0 --force --output json ``` `--list` and `--version` are mutually exclusive. The target must be older than the current version and at least the CLI version floor. System-package installations must use their package manager. Releases before 1.10.4 are pinned by disabling their legacy automatic-update setting. The JSON result reports this as `automatic_updates_disabled`. ## Rollback ```bash theme={null} observal self rollback --force --output json ``` Rollback is available only for standalone binary installations with `~/.observal/bin/observal.prev`. It acquires the same version-change lock as upgrade and downgrade, copies the backup to a temporary file, restores executable permissions, and atomically replaces the current binary. Package-managed installations must install their previous version through the package manager. ## Installation safety Upgrade and downgrade: 1. Acquire a process lock so version changes cannot overlap. 2. Use the detected installation method. 3. Verify standalone binary downloads against the published SHA-256 checksum. 4. Replace standalone binaries atomically and retain the previous binary for rollback. 5. Execute the installed binary and verify that it reports the requested version. Human mode may explicitly confirm an unsigned binary. JSON mode never prompts and refuses standalone installation when the release has no published checksum. ## Exit codes | Code | Meaning | | - | - | | 5 | Rollback backup not found | | 6 | Package-managed install or another version change holds the lock | | 7 | Invalid version, direction, conflicting mode, or missing `--force` in JSON mode | | 9 | GitHub, download, installer, checksum, verification, or filesystem failure | ## Related * [`observal server`](/cli/server): manage server stack versions separately * [Upgrades](/self-hosting/upgrades): server and CLI upgrade procedures * [Environment variables](/reference/environment-variables): update-check settings # observal server Source: https://docs.observal.io/cli/server Manage local Observal deployments. The lifecycle commands operate the embedded PostgreSQL, ClickHouse, Redis, and API processes. Upgrade, rollback, and version commands operate a local Docker Compose deployment. Local filesystem, process, Docker, and database access are the authorization boundary. These commands do not require a reachable Observal API or an API role. ## Command summary | Command | Target | Purpose | | - | - | - | | `start` | Embedded | Start dependencies and the API | | `stop` | Embedded | Stop all services | | `restart` | Embedded | Stop and start services | | `status` | Embedded | Report service health and ports | | `logs` | Embedded | Read or follow service logs | | `install` | Embedded | Install verified dependency binaries | | `config` | Embedded | Show local paths and ports | | `reset` | Embedded | Delete database data and generated secrets | | `upgrade` | Docker | Back up PostgreSQL and replace images | | `rollback` | Docker | Restore PostgreSQL and the prior image version | | `versions` | Docker | List image versions and managed backups | | `migrate` | Databases | Move PostgreSQL registry and ClickHouse telemetry data | ## Embedded lifecycle Start in the foreground: ```bash theme={null} observal server start ``` Start for automation: ```bash theme={null} observal server start --background --output json ``` JSON start and restart require `--background` because foreground mode remains attached until shutdown. ```bash theme={null} observal server restart --background --output json observal server stop --output json ``` `start` accepts `--port/-p` and `--host`. When the default API port is occupied, it tries the documented local fallback ports and reports the selected port. An explicitly selected occupied port is a conflict. Background start and restart verify all four final service states before returning success; stop verifies that none remains running. Their JSON results include the verified `services` list, and an unverified transition exits with code 9 plus observed state in `error.result`. Startup performs these steps in order: 1. Installs embedded dependencies when missing. Downloads require a published SHA-256 checksum and archives reject links and path traversal. 2. Starts PostgreSQL, ClickHouse, and Redis. 3. Applies PostgreSQL and ClickHouse migrations. Migration failures stop startup; the CLI never stamps a failed PostgreSQL schema as current. 4. Starts the API. 5. Bootstraps a local admin only on a fresh embedded server and persists the real access and refresh tokens. It never writes placeholder credentials or an API key. 6. Attempts telemetry hook installation. Optional hook failures are explicit warnings. ## Status and configuration ```bash theme={null} observal server status --output json observal server config --output json ``` Status is a finite diagnosis command. It exits successfully when checks run, including when `healthy` is false. ```json theme={null} { "healthy": false, "services": [ {"service": "postgres", "status": "running", "port": 5480}, {"service": "clickhouse", "status": "stopped", "port": 8124}, {"service": "redis", "status": "running", "port": 6380}, {"service": "api", "status": "stopped", "port": 8000} ] } ``` Configuration output contains paths and ports only. It never returns generated secrets. ## Logs Read a bounded snapshot: ```bash theme={null} observal server logs --output json observal server logs api --lines 200 --output json ``` Follow one service as JSON Lines: ```bash theme={null} observal server logs api --follow --output json ``` JSON follow requires one service so every event has an unambiguous `service` field. Valid services are `postgres`, `clickhouse`, `redis`, and `api`. ## Install and reset ```bash theme={null} observal server install --output json observal server install --upgrade --output json ``` Reset deletes embedded database directories and the generated server secret. It does not delete CLI configuration, downloaded binaries, logs, or unrelated files. Human mode confirms. JSON mode requires `--force`: ```bash theme={null} observal server reset --force --output json ``` Deletion is confined to the managed embedded data directory. ## Docker upgrade Preview an upgrade: ```bash theme={null} observal server upgrade --dry-run --output json ``` Apply one non-interactively: ```bash theme={null} observal server upgrade --version 1.2.3 --force --output json ``` An upgrade validates the target version and image, acquires the server upgrade lock, creates a managed PostgreSQL backup unless `--skip-backup` is set, pulls images, atomically updates `OBSERVAL_VERSION`, recreates containers, and runs the configured health check. A failed health check requests the previous image version again and returns an unavailable error. JSON mutation requires `--force`; dry run does not. ## Docker rollback ```bash theme={null} observal server rollback --force --output json observal server rollback \ --from-backup ~/.observal/backups/v1.2.2-20260521T120000 \ --force --output json ``` Rollback accepts only backup directories under the managed backup root. It restores PostgreSQL, atomically restores the image version, recreates containers, and checks health. **ClickHouse telemetry is not restored by this command.** JSON and human results state `clickhouse_restored: false`. Use [`observal server migrate`](/cli/migrate) for ClickHouse export and import. ## Docker versions ```bash theme={null} observal server versions --output json ``` The result distinguishes the current version, available GHCR images, and local PostgreSQL backups. Failure to query GHCR is reported as unavailable rather than as an empty registry. ## Error and output contract All finite commands accept `--output table|json`. JSON success writes one document to stdout. Failures leave stdout empty and write one categorized error to stderr. Log following is the only JSON Lines stream. | Code | Category | Typical server cause | | - | - | - | | 2 | Usage | Invalid option or missing required argument | | 4 | Permission | Unreadable local state or backup outside the managed root | | 5 | Not found | Missing logs, Compose deployment, image, or backup | | 6 | Conflict | Busy port, active upgrade lock, or existing destination | | 7 | Validation | Missing JSON confirmation or invalid version | | 9 | Unavailable | Dependency, Docker, migration, health, or network failure | ## Related * [Database migration](/cli/migrate) * [Self-hosted upgrades](/self-hosting/upgrades) * [Backup and restore](/self-hosting/backup-and-restore) * [`observal self`](/cli/self), for CLI binary versions # observal share Source: https://docs.observal.io/cli/share Create and consume expiring links for version-pinned Agents installed in a repository. Share links preserve Registry authorization: recipients must sign in to the same Observal deployment and can only see and pull Agents available in their current public or teamspace scope. ## Create a share Run the command in a repository and select Agents interactively: ```bash theme={null} observal share ``` Share every tracked project Agent: ```bash theme={null} observal share --all --expires-days 7 ``` Select explicit installed Agents: ```bash theme={null} observal share --agent platform/reviewer --agent platform/github-helper ``` The printed URL is not a web page. Open it only with `observal share open`; the web UI has no share route yet. The bare token also works if the URL's host does not match your configured server (`deployment.frontend_url`). Links expire after 7 days by default. The allowed range is 1 through 30 days. Only project-scoped Agents recorded in the current Registry section of `~/.observal/lockfile.json` are candidates. The repository path, Git remote, harness names, configuration, prompts, and credentials are not uploaded. ## Inspect candidates ```bash theme={null} observal share candidates observal share candidates --dir ./service --output json ``` Discovery checks every harness section in the lockfile and deduplicates identical Agent/version pairs. ## Open and pull ```bash theme={null} observal share open 'https://observal.example/shares/agents/' ``` Observal validates that the URL belongs to the configured deployment, fetches the manifest using the configured bearer token, and displays its creator and accessible Agents. It does not contact the host contained in the supplied URL. After selection and confirmation, the command asks for a target harness and delegates each installation to `observal agent pull`. Use `--no-pull` to inspect without installing: ```bash theme={null} observal share open --no-pull ``` Only currently accessible, approved Agent versions are returned. Inaccessible entries are reported as a count without exposing their identity. ## Revoke The creator or an administrator can revoke a link immediately: ```bash theme={null} observal share revoke observal share revoke --yes --output json ``` Expired and revoked links cannot be opened. ## Security properties * Public links contain a 256-bit random opaque token, not Agent IDs or repository metadata. * The server stores only a SHA-256 hash of the token. * Links do not grant permissions; Registry visibility is evaluated on every open. * Creation validates every Agent and exact semantic version in one transaction. * URLs with credentials, query strings, fragments, foreign origins, control characters, or invalid paths are rejected. * Pull commands use argument arrays rather than a shell, preventing values from becoming shell syntax. # observal registry skill Source: https://docs.observal.io/cli/skill Submit, browse, review, install, and safely update versioned skill folders, as well as Git-backed and historical single-file skills. See the [complete skill folder workflow](/skill-folder-workflow) for the owner-to-reviewer-to-consumer journey and rollout requirements. ## Commands | Command | Description | | - | - | | `submit` | Submit a Git-backed or registry-direct skill | | `list` | List approved visible skills | | `my` | List your skills across all statuses | | `show` | Show one skill, visible source provenance, and public fork count | | `fork-listing` | Copy an approved skill release into a separate, independently owned draft | | `install` | Write a skill into a supported harness | | `edit` | Edit a draft, pending, or rejected skill | | `fork` | Create a new draft from the exact reviewed direct release | | `import-folder` | Import a complete local folder as a successor to a reviewed Git or historical direct release | | `replace-files` | Replace a draft's complete file tree, binding to its version UUID and observed revision | | `withdraw` / `rebase` | Withdraw a pending version or rebase a saved draft against a newer reviewed version | | `export` | Export an exact version to a new local directory | | `backups list` / `restore` / `prune` | Inspect, restore, or explicitly prune verified managed-folder backups | | `archive` | Archive an approved skill | | `unarchive` | Restore an archived skill | | `transfer-owner` | Transfer ownership | | `co-authors` | List, add, or remove co-authors | Every command that returns structured data supports table and JSON output. Archive, restore, and ownership transfer require the confirmation bypass in JSON mode. ## Submit Git-backed skills require a Git URL. Registry-direct skills can store `SKILL.md`, optional script resources, or a complete bounded folder with scripts, templates, binary assets and executable modes. Folder drafts are versioned; saving a draft does **not** submit it for review. Use an explicit `fork` or `import-folder` to create a successor to an approved release; never use `submit --from-dir` to silently overwrite one. ```bash theme={null} observal registry skill submit \ --skill-md ./SKILL.md \ --git-url https://github.com/acme/review-skill \ --name review-skill \ --description "Review code" \ --task-type code-review \ --output json observal registry skill submit \ --skill-md ./SKILL.md \ --delivery-mode registry_direct \ --name review-skill \ --description "Review code" \ --task-type code-review \ --output json ``` For a new complete folder, use `observal registry skill submit --from-dir ./my-skill --name my-skill --description "My skill"`. To advance an approved listing, use `fork` (reviewed direct base) or `import-folder` (reviewed Git/historical direct base), then `observal registry skill replace-files my-skill --version-id UUID --from-dir ./my-skill --revision OBSERVED_REVISION` as needed. The [folder workflow](/skill-folder-workflow) covers exact-base selection, sensitive/excluded file acknowledgments, review, and immutable releases. JSON mode never prompts; supply all required inputs explicitly. Valid task types are `code-review`, `code-generation`, `testing`, `documentation`, `debugging`, `refactoring`, `deployment`, `security-audit`, `performance`, and `general`. ## List and show ```bash theme={null} observal registry skill list --task-type code-review --output json observal registry skill list --harness claude-code --output json observal registry skill my --output json observal registry skill show acme/review-skill --output json ``` Row numbers are scoped to the latest Skill list. Empty lists clear previous Skill row references. ## Fork ```bash theme={null} observal registry skill fork-listing acme/review-skill --name my-review-skill --output json observal registry skill fork-listing acme/review-skill --team platform --visibility team --output json observal registry skill edit FORK_UUID --description 'Customized review' --output json observal registry skill submit --submit FORK_UUID --output json ``` The source listing and selected version must be approved; the copy begins as an independent draft. Inline `registry_direct` content is copied, but its validation flag resets for the normal submission path. Use the returned UUID to edit and submit without affecting the source. Only publicly approved forks appear in fork counts; private provenance is never exposed to viewers who lack source access. ## Install ```bash theme={null} observal registry skill install acme/review-skill --harness claude-code --scope user --output json observal registry skill install acme/review-skill --harness pi --scope project --output json observal registry skill install acme/review-skill --harness pi --no-write --output json observal registry skill install acme/review-skill --harness pi --raw ``` JSON output performs the installation unless no-write is selected. It reports `write_performed` and `installed_path`. Raw mode emits only the generated config and performs no write. Lockfile state is recorded only after the skill content is written successfully. A complete folder install requires the server's default-off delivery gate, a compatible client, and a POSIX filesystem (Linux/macOS or WSL on its Linux filesystem). Native Windows managed folder installation, upgrades, backup restoration, and pinned Agent folder pulls are not supported; they fail closed rather than attempting an unsafe partial swap. The other CLI workflows and existing Git/single-file installations are not subject to this folder-swap restriction. Pin an approved `--version` for reproducible installs; omitting it follows the latest release. A managed folder is never replaced silently: first use `--check-upgrade` for a no-write preview, then `--upgrade` to verify every installed byte and mode, retain a private backup and update its receipt. `--backup-root DIR` must be private, ignored by Git when inside a worktree, outside skill discovery roots, and on the target filesystem. Cross-mode Git/single-file installs cannot erase existing folder receipts. The command fails if the harness lacks skill support, the source cannot be installed, the destination is unowned or locally edited, or installed state cannot be recorded. See [verified updates and recovery](/skill-folder-workflow#verified-managed-update-and-recovery) before restoring a backup. ## Edit ```bash theme={null} observal registry skill edit acme/review-skill --description "Updated" --output json observal registry skill edit acme/review-skill --from-file updates.json --output json ``` Edit-lock conflicts preserve conflict exit code 6. Invalid fields and files use the shared validation, not-found, permission, and unavailable categories. ## Related * [`observal registry`](/cli/registry): complete registry reference * [`observal agent`](/cli/agent): attach skills to agents # observal doctor support Source: https://docs.observal.io/cli/support Generate and inspect redacted diagnostic support bundles. Both workflows support `--output table|json`. Bundle generation writes a `.tar.gz` archive as its intended side effect. JSON mode returns archive metadata and never prompts. ## Generate a bundle ```bash theme={null} observal doctor support bundle --output json ``` Choose the archive path with the hard-renamed `--file/-f` option: ```bash theme={null} observal doctor support bundle \ --file /tmp/observal-support.tar.gz \ --logs-since 2h \ --no-include-system \ --output json ``` The former archive-path meaning of `--output/-o` has been removed. `--output/-o` now consistently selects `table` or `json`. ### Options | Option | Purpose | | - | - | | `--file`, `-f` | Archive path; defaults to a timestamped file in the current directory | | `--logs-since` | Positive duration up to 30 days, such as `30m`, `2h`, or `1d12h` | | `--include-system`, `--no-include-system` | Include or omit local system metrics | | `--force`, `--yes` | Overwrite an existing archive and accept the size warning | | `--output`, `-o` | Human or JSON result | Existing archives prompt in human mode. JSON mode fails with a conflict unless `--force` or `--yes` is present. JSON returns: ```json theme={null} { "path": "/tmp/observal-support.tar.gz", "size_bytes": 43120, "remote_status": "collected", "warnings": [], "collector_results": {}, "redaction_counts": {} } ``` ## Collection behavior Remote collectors use `POST /api/v1/support/collect` and require an administrator account. They provide: * Application and migration versions * PostgreSQL, ClickHouse, and Redis health * Allowlisted configuration * Aggregate table counts * Error fingerprints * Recent redacted log records Local collectors provide allowlisted configuration processing and optional OS, CPU, memory, disk, and container-runtime details. Remote collection is optional by design. Authentication, permission, version, network, or response failures are recorded in `remote_status` and `warnings`; local collection continues. This fallback is never silent. Individual collector failures are retained in the manifest and JSON result. If no collector produces data, the command exits with code 9 and does not create an archive. ## Privacy and integrity Support bundles contain aggregate diagnostics, not customer database rows or trace payloads. Protections include: * Configuration allowlisting before archive creation * Redaction of passwords, tokens, API keys, private keys, authorization values, URL credentials, JWTs, AWS access keys, and high-entropy strings * Redaction of collector error text before it enters the manifest * Rejection of unknown collector names and unsafe archive paths * Hashed host identity rather than a raw hostname * SHA-256 inventory entries for every archived file * Atomic archive replacement with mode `0600` The manifest records collector status, redaction counts, file sizes, and hashes. Review every bundle before sharing it because logs and deployment metadata can still be operationally sensitive. ## Size budget The uncompressed budget is 100 MB. Human mode asks before exceeding it. JSON mode requires `--force` and never prompts. ## Inspect a bundle ```bash theme={null} observal doctor support inspect ./observal-support.tar.gz --output json ``` Show one regular file: ```bash theme={null} observal doctor support inspect ./observal-support.tar.gz \ --show health/postgres.json \ --output json ``` JSON returns the manifest, safe regular-file inventory, schema warnings, and optional shown content: ```json theme={null} { "manifest": {"bundle_schema_version": "1"}, "files": [{"path": "health/postgres.json", "size_bytes": 91}], "warnings": [], "shown": {"path": "health/postgres.json", "content": "{\"status\":\"ok\"}"} } ``` Inspection never extracts files. It rejects traversal paths, links requested through `--show`, malformed or oversized manifests, and shown files larger than 1 MB. Unsafe archive members are omitted and reported as warnings. ## Exit codes | Code | Meaning | | - | - | | 5 | Bundle or requested member not found | | 6 | Destination archive already exists | | 7 | Invalid duration, unsafe archive, malformed manifest, oversized display, or missing non-interactive confirmation | | 9 | No diagnostic data or archive write failure | ## Related * [`observal doctor`](/cli/doctor): diagnose and repair telemetry instrumentation * [`observal ops telemetry status`](/cli/ops): check telemetry flow * [Troubleshooting](/self-hosting/troubleshooting): deployment diagnosis # observal team Source: https://docs.observal.io/cli/team Create and govern teamspaces, members, visibility reviews, join requests, and private-team invitations. Every leaf command supports `--output table|json`. JSON success output contains no Rich text or prompts. JSON failures leave stdout empty and write one categorized error object to stderr. ## Commands | Command | Purpose | | - | - | | `list` | List the signed-in user's teamspaces | | `show` | Show a teamspace and its members | | `create` | Create a teamspace | | `claim-personal` | Claim or return the caller's personal teamspace | | `delete` | Permanently delete a teamspace | | `leave` | Leave a teamspace | | `visibility set` | Change visibility or request public review | | `visibility list-requests` | List pending public visibility requests | | `visibility approve` | Approve pending public visibility | | `visibility reject` | Reject pending public visibility | | `request join` | Request membership | | `request list` | List a teamspace's join requests | | `request mine` | View the caller's request history and status | | `request withdraw` | Withdraw the caller's pending request | | `request approve` | Approve a pending request | | `request reject` | Reject a pending request | | `members list` | List members | | `members add` | Add a member or update a role | | `members remove` | Remove a member | | `invite create` | Create a private-team invitation | | `invite list` | List invitations | | `invite revoke` | Revoke an invitation | | `invite preview` | Preview an invitation token | | `invite request` | Use a token to request access | | `invite delete` | Delete an unused invitation | | `invite requests` | List requests associated with an invitation | Team references may be UUIDs, handles, or `@handle`. Unknown teamspaces use not-found exit code 5. ## List, show, and claim ```bash theme={null} observal team list --output json observal team list --all --output json observal team show platform-tools --output json observal team claim-personal --output json ``` `list` returns the standard `items`, `total`, `page`, and `page_size` envelope. The default includes teamspaces where the user is a member. `--all` requests all teamspaces visible to the caller. Empty results use `items: []` and `page_size: 0`. `show` returns a combined result: ```json theme={null} { "team": { "id": "11111111-1111-1111-1111-111111111111", "name": "Platform Tools", "handle": "platform-tools", "role": "owner" }, "members": [] } ``` `claim-personal` is idempotent. It creates or returns the caller's one private personal teamspace and returns the direct Team object. ## Create and visibility ```bash theme={null} observal team create 'Platform Tools' \ --handle platform-tools \ --description 'Internal tooling' \ --visibility private \ --output json observal team visibility set platform-tools public --output json ``` Visibility is `public` or `private`. Creating a public teamspace or setting a private teamspace to public submits a review request. Until approval, the response reports `visibility: "private"` and `visibility_request_status: "pending"`. Create and visibility JSON return the direct Team object. Reviewers and deployment admins manage pending public visibility requests: ```bash theme={null} observal team visibility list-requests --output json observal team visibility approve platform-tools --output json observal team visibility reject platform-tools \ --reason 'Add a public description' \ --output json ``` The list returns the standard list envelope. Approve and reject return the direct Team object. Approval makes the teamspace public and revokes its private invitation links. A rejection reason is optional and accepts up to 500 characters. ## Delete and leave ```bash theme={null} observal team delete platform-tools --yes --output json observal team leave platform-tools --yes --output json ``` Delete is permanent. Leave removes only the caller's membership. The last owner cannot leave. Human mode prompts unless `--yes` is supplied. JSON mode never prompts and requires `--yes`. Both endpoints currently return an empty JSON object on success. ## Join requests Request access to a visible teamspace: ```bash theme={null} observal team request join platform-tools \ --message 'I maintain deployments' \ --output json ``` The message is optional and limited to 500 characters. JSON returns the created join request. View your own status or withdraw the sole pending request: ```bash theme={null} observal team request mine platform-tools --output json observal team request withdraw platform-tools --yes --output json ``` `request mine` returns the standard list envelope with requests ordered newest first. `request withdraw` finds the caller's pending request and marks it cancelled. Human mode prompts unless `--yes` is supplied. JSON mode requires `--yes` and returns an empty object. Owners and deployment admins can list and decide requests: ```bash theme={null} observal team request list platform-tools --status pending --output json observal team request approve platform-tools @alice --output json observal team request reject platform-tools bob@example.com \ --reason 'Use the SRE teamspace' \ --output json ``` Valid status filters are `pending`, `approved`, `rejected`, and `cancelled`. Approve and reject select a pending request by exact email or case-insensitive username. A missing pending request uses not-found exit code 5. The list returns the standard list envelope. Join, approve, and reject return the direct join-request object. ## Members ```bash theme={null} observal team members list platform-tools --output json observal team members add platform-tools alice@example.com --role reviewer --output json observal team members add platform-tools @bob --role owner --output json observal team members remove platform-tools @bob --yes --output json ``` Roles are `member`, `reviewer`, and `owner`. Adding an existing member updates the role. The last owner cannot be removed. Member list returns the standard list envelope. Add returns the saved member. Remove currently returns an empty object. Human remove prompts unless `--yes` is supplied. JSON remove requires `--yes`. ## Private-team invitations Create and list invitations: ```bash theme={null} observal team invite create platform-tools \ --name onboarding \ --expires-days 30 \ --max-uses 20 \ --output json observal team invite list platform-tools --output json ``` `--expires-days` accepts 1 through 365. `--max-uses` accepts 1 through 10,000 or may be omitted for no use limit. Invite names accept 1 through 100 characters. Create returns the direct invitation object, including the one-time token and URL. Treat both as secrets. List returns the standard list envelope. States include active, expired, exhausted, and revoked. A recipient can preview the token, then submit an owner-reviewed access request: ```bash theme={null} observal team invite preview INVITE_TOKEN --output json observal team invite request INVITE_TOKEN \ --message 'I am joining the deployment rotation' \ --output json ``` `invite preview` does not mutate membership or request state. `invite request` does not grant membership. It creates a pending join request that an owner must approve. Neither command prints the token in human output or errors. Owners and admins can inspect invitation usage: ```bash theme={null} observal team invite requests \ platform-tools \ 550e8400-e29b-41d4-a716-446655440000 \ --output json ``` The command returns the standard list envelope for requests associated with that invitation. Empty results use `items: []`. Revoke an invitation while retaining its audit history: ```bash theme={null} observal team invite revoke \ platform-tools \ 550e8400-e29b-41d4-a716-446655440000 \ --yes \ --output json ``` Delete an invitation only when it has no uses or request history: ```bash theme={null} observal team invite delete \ platform-tools \ 550e8400-e29b-41d4-a716-446655440000 \ --yes \ --output json ``` The invite ID must be a UUID. Human mode prompts unless `--yes` is supplied. JSON mode requires `--yes`. Revoke returns the invitation object. Delete returns an empty object. ## Exit codes | Code | Meaning | | - | - | | 3 | Authentication required or failed | | 4 | Membership, owner, reviewer, or admin permission denied | | 5 | Teamspace, invitation, member, or request not found | | 6 | Handle, membership, owner, visibility, invitation, or request state conflict | | 7 | Invalid visibility, role, status, UUID, text length, token, or missing JSON confirmation | | 8 | Rate limit reached | | 9 | Server unavailable | | 10 | CLI and server version mismatch | ## Related * [`observal inbox`](/cli/inbox): view request and visibility decisions * [`observal agent`](/cli/agent): publish Agents to a teamspace * [`observal registry`](/cli/registry): publish components to a teamspace # Code Review Standard Source: https://docs.observal.io/code-review Code review protects Observal users, contributors, and maintainers. Its purpose is not merely to make a pull request pass CI. Reviewers are responsible for deciding whether a change is correct, secure, understandable, appropriately tested, and safe to maintain. This standard applies to every pull request merged into a protected branch. The depth of review should match the risk of the change, but the approval and merge requirements apply even to small changes. ## Required reviewers Every pull request requires: * Approval from at least two eligible reviewers. An eligible reviewer is a project collaborator whose approval GitHub counts toward the protected branch. * Approval from a code owner when the changed paths have an applicable entry in `CODEOWNERS`. A code-owner approval may count as one of the two required approvals. * Additional review from someone qualified in any area that the assigned reviewers cannot confidently assess, such as authentication, authorization, cryptography, privacy, database migrations, release infrastructure, accessibility, or concurrency. Anyone may comment and provide useful review feedback. Reviewers who are not eligible approvers should state that their review is advisory. A reviewer covering only part of a pull request must identify the files or concerns they reviewed and must not imply approval of the remainder. Author self-approval is prohibited. An approval from a co-author or anyone who contributed commits to the pull request does not count toward the two independent approvals. ## Reviewer procedure ### 1. Understand the change Before reviewing individual lines: 1. Read the pull request description, linked issue, prior design discussion, and testing evidence. 2. Confirm that the proposed behavior is wanted and that the pull request addresses the stated problem. 3. Check that the change is focused. Unrelated refactors, formatting, generated files, or dependency updates should be separated unless they are necessary for the change. 4. Identify high-risk areas and request a qualified reviewer early when needed. If the purpose, expected behavior, or risk is unclear, ask for clarification before approving. ### 2. Review the complete change Review every human-written line in the diff and enough surrounding code to understand its effect. Do not rely only on the changed lines. Trace callers, data flow, failure paths, configuration, migrations, and user-visible behavior where relevant. Generated files may be sampled when their source and generation command are clear. Lockfiles still require review for unexpected packages, sources, versions, and integrity data. Automated checks support human review but do not replace it. A green pipeline cannot detect every authorization error, privacy leak, race, incompatible interface, or incorrect product decision. ### 3. Validate the evidence Confirm that the pull request explains how the change was tested. Run focused tests or reproduce the behavior when the risk, novelty, or user impact warrants independent validation. Ask for screenshots, recordings, logs, benchmarks, migration output, or a demonstration when code inspection alone is insufficient. A reviewer must not approve code they do not understand. Ask questions or bring in another reviewer instead. ### 4. Give actionable feedback Use **Request changes** for blocking findings. Explain the risk and the condition that must be satisfied. Mark optional improvements clearly as non-blocking or as a nit. Prefer comments about correctness, risk, maintainability, and documented project conventions over personal style preferences. Suggest a follow-up issue only when the current change is safe and complete without the follow-up. The author must acknowledge every blocking thread. A thread may be resolved when the change is made or when the author and reviewer agree, with the reasoning recorded in the pull request. ### 5. Review the latest revision Review the final diff, not only the first version submitted. After a material push, prior approvals are stale. All required approvers must review the delta and approve or explicitly reaffirm approval on the latest material revision. A push is material when it changes behavior, interfaces, security or privacy properties, data handling, dependencies, migrations, build or release logic, test strategy, or the agreed scope. Typographical and other clearly non-behavioral corrections are not normally material, but a reviewer may request another review whenever the impact is uncertain. ## What every review must cover ### Correctness and design Confirm that: * The implementation satisfies the documented requirements and preserves relevant invariants. * Error handling is explicit and does not hide failed operations, partial writes, or data loss. * Edge cases, empty and malformed inputs, retries, concurrency, idempotency, and resource cleanup are handled where relevant. * The design follows existing project architecture and uses an existing helper or platform feature where appropriate. * The change does not introduce unnecessary abstraction, configuration, dependencies, dead code, or speculative flexibility. * Performance, availability, and operational costs are reasonable for expected production use. ### Security and privacy Treat all external input and cross-tenant data as untrusted. Confirm that: * Authentication and authorization are enforced at the server boundary, not only in the user interface. * Access checks cover object ownership, team boundaries, administrative roles, and indirect lookup paths. * Inputs are validated and are safe from injection, path traversal, request forgery, unsafe redirects, and insecure deserialization. * Secrets, tokens, credentials, personal data, and telemetry are not exposed through logs, errors, URLs, browser storage, analytics, or generated artifacts. * Data collection, retention, export, deletion, and redaction remain consistent with the documented privacy model. * Cryptography and security controls use established project mechanisms and secure defaults. * New dependencies, actions, container images, and downloaded tools are necessary, integrity-pinned where supported, and from trustworthy sources. Do not discuss an undisclosed vulnerability in a public pull request. Follow the private process in the [Security Policy](https://github.com/Observal/Observal/blob/main/SECURITY.md). ### Tests and CI Confirm that: * New behavior has tests at the smallest effective level, with integration or end-to-end coverage where boundaries or user workflows require it. * A bug fix includes a regression test that fails without the fix whenever practical. * Tests assert observable behavior, exercise failure paths, and would detect a real regression. * Tests are deterministic, isolated from real user data and external services, and no more complex than necessary. * Required CI checks pass for the final material revision. Flaky, skipped, cancelled, or unrelated failures must be investigated and documented, not silently ignored. ### Compatibility and data safety Confirm that: * Public APIs, CLI commands, configuration, stored data, deployment files, supported platforms, and harness integrations remain compatible unless an intentional change is approved and documented. * Breaking changes include a clear rationale, user migration path, release note, and coordinated updates to every in-repository caller. * Database migrations use the repository migration tooling, have been inspected, preserve existing data, and have a safe upgrade and recovery plan. * Changes that cross versions, workers, caches, queues, or rolling deployments account for mixed-version operation where applicable. ### Documentation and user experience Confirm that: * User-visible behavior, APIs, commands, configuration, setup, and operational procedures are documented in the same pull request. * CLI syntax changes are reflected in bundled skill documentation. * User-facing changes include the required changelog entry and accurate screenshots or other evidence when applicable. * Interfaces remain accessible, understandable, responsive, and consistent with established design patterns. * Comments explain non-obvious intent and trade-offs rather than restating the code. ### Licensing and provenance Confirm that: * New files have the correct SPDX copyright and license headers. * The contributor has satisfied the CLA and followed the [AI Policy](https://github.com/Observal/Observal/blob/main/AI_POLICY.md). * Copied or adapted material is compatible with Apache-2.0, properly attributed, and accompanied by any required notices. * New dependencies, assets, generated code, and vendored files have acceptable licenses and documented origins. ### Scope and maintainability Confirm that: * The pull request contains one coherent change and no unrelated cleanup. * The implementation is readable by maintainers who did not author it. * Temporary code, debugging output, compatibility workarounds, and obsolete paths are removed unless explicitly justified. * The pull request description, commit history, and documentation accurately describe the final implementation. ## Approval and merge requirements A pull request is acceptable for merge only when all of the following are true: * Two eligible reviewers have approved the latest material revision. * A code owner has approved where `CODEOWNERS` applies. * No reviewer has an outstanding **Request changes** review. * Every review thread is resolved with its outcome recorded. * All required CI checks pass on the mergeable revision. * Required tests, documentation, changelog entries, screenshots, migration evidence, SPDX headers, and license information are present. * The CLA and other repository policy checks pass. * The pull request is focused, understandable, and ready to operate without relying on an unspecified follow-up. The person merging the pull request performs a final gate check. Merge permission is not permission to bypass this standard. ## When to request changes or reject a pull request A reviewer must withhold approval and request changes when any blocking issue remains, including: * Incorrect behavior, an unhandled failure mode, data-loss risk, or an unresolved security or privacy concern. * Missing or ineffective tests for behavior that can reasonably be tested. * A breaking or operationally unsafe change without an approved migration and recovery plan. * Missing required documentation, licensing information, attribution, or policy compliance. * Scope or complexity that prevents a reliable review. The reviewer may require the pull request to be split or redesigned. * A material revision that has not been reviewed by the required approvers. A pull request should be rejected or closed when it: * Conflicts with an established project decision or does not solve an accepted project need. * Duplicates existing work without a reason to replace it. * Introduces malicious, deceptive, obfuscated, or unverifiable behavior. * Contains code or assets that the contributor cannot legally license to the project. * Violates the Code of Conduct, CLA, AI Policy, responsible disclosure process, or other repository policy. * Remains abandoned after maintainers have clearly requested the information or changes needed to continue. Reviewers should explain rejection clearly and respectfully. Rejection is a decision about the proposed change, not the contributor. ## Review conduct and disagreements Follow the [Code of Conduct](https://github.com/Observal/Observal/blob/main/CODE_OF_CONDUCT.md). Be specific, constructive, and proportionate to risk. Authors should not interpret questions as approval, and reviewers should not use approval as leverage for unrelated preferences. Resolve disagreements using documented requirements, reproducible evidence, and project architecture. If consensus cannot be reached, ask another domain expert or maintainer to decide and record the decision in the pull request. Do not leave a pull request indefinitely blocked by an unstated concern. Urgent and embargoed security changes may be reviewed privately by a restricted group, but independent review, testing, and approval are not waived. ## Further reading This standard was informed by established review practices from: * [Google Engineering Practices: What to look for in a code review](https://google.github.io/eng-practices/review/reviewer/looking-for.html) * [GitLab Code Review Guidelines](https://docs.gitlab.com/development/code_review/) * [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) * [Django Contribution Checklist](https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/submitting-patches/#contribution-checklist) * [OpenSSF Source Code Management Platform Configuration Best Practices](https://best.openssf.org/SCM-BestPractices/) # Core concepts Source: https://docs.observal.io/core-concepts/index The vocabulary you need to be productive with Observal. ## The registry The registry stores users, agents, components, review state, and alert rules in PostgreSQL. Session events and aggregates are stored separately in ClickHouse. See [Session tracking and reconciliation](/core-concepts/session-tracking) for the complete session data flow. ## Registry components Six component types are available. Agents bundle the other five. | Type | What it is | | - | - | | **Agent** | A complete, installable AI agent. Bundles MCP servers, skills, hooks, prompts, and sandboxes into one YAML. | | **MCP Server** | A [Model Context Protocol](https://modelcontextprotocol.io/) server, the tools an agent can call. | | **Skill** | A portable instruction package agents load on demand. | | **Hook** | A lifecycle callback that runs on session start, tool use, session end, and other supported events. | | **Prompt** | A named, parameterized prompt template with variable substitution. | | **Sandbox** | A Docker execution environment for running code the agent generates. | Anyone can publish. Admin review controls what appears in the public listing, but your own items are usable immediately without approval. ## Canonical identity An agent or component has a canonical `namespace/slug` identity. The namespace is the owner's username. The slug is stable within that namespace. Display names can change without changing the canonical identity. UUIDs are the stable reconciliation key. The server supplies canonical namespace, slug, display name, qualified name, and review status. Installed versions in the local lockfile remain local pins. ## Versions and locks Agents and components are versioned. An approved version never changes. Every agent version pins each of its components to one exact version, recorded with a content digest. Installing that agent version always installs those component versions, even after newer ones are approved. Agent versions released before pinning are the exception: a component they never locked installs at its latest approved release, with a warning. Only the agent's author moves them, by releasing a new agent version, which goes through review like any other. Installs are pinned too. The first `observal agent pull` in a project records the agent version it installed in `observal.lock`, a file meant to be committed. Later pulls, by anyone in that project, install the same agent version until someone moves it with `--upgrade` or `--version`. `~/.observal/lockfile.json` is each machine's record of what it has installed where. `--strict` refuses any install that does not match its lock. See [`observal agent pull`](/cli/pull#pinned-versions) and [`observal agent release`](/cli/agent#release-and-versions). ## Deployment mode SSO-only access is controlled by `deployment.sso_only`: | Mode | Self-registration | Bootstrap | Auth | | - | - | - | - | | `deployment.sso_only=false` (default) | Yes | Yes, a fresh server creates an admin on first login | Email and password or API key | | `deployment.sso_only=true` | No | No | SSO only | Most self-hosters use `deployment.sso_only=false`. ## Next * [Session tracking and reconciliation](/core-concepts/session-tracking) * [Use cases](/use-cases) # Session tracking and reconciliation Source: https://docs.observal.io/core-concepts/session-tracking Observal turns local harness conversations into indexed session events. This page explains the complete path from a harness source through durable delivery, server parsing, reconciliation, and recovery. ## Data model ### Session A session is one harness conversation or task identified by `session_id`. Observal records the harness, authenticated user, project, agent attribution, and source position with each record. ### Source record A source record is one complete item emitted by the harness. For a JSONL source, it is one non-empty, newline-terminated line. For a native source, the extension creates an equivalent stable record before delivery. ### Event The server parser converts each source record into an event such as a user prompt, assistant response, tool call, tool result, token update, lifecycle hook, error, or subagent event. Available detail depends on the harness source. ### Aggregate `session_stats_agg` summarizes event counts, prompts, tool calls, tool results, token totals, models, harnesses, agent attribution, and timing. Dashboards and insights query these aggregates. ## End-to-end flow ```mermaid theme={null} flowchart TD source["Harness source"] wake["Hook, extension, or reconcile"] adapter["Harness adapter"] outbox["Durable local outbox"] ingest["Session ingest API"] records["Raw session records"] parser["Harness parser"] events["Parsed events and aggregates"] checkpoint["Contiguous server checkpoint"] source --> adapter wake --> adapter adapter --> outbox outbox --> ingest ingest --> records records --> parser parser --> events ingest --> checkpoint checkpoint --> outbox ``` The normal pipeline has seven stages: 1. The harness writes a JSONL transcript or exposes messages through a native API. 2. A hook, extension event, or `observal reconcile` wakes the exporter. 3. The harness adapter discovers the source and reads only complete records after the current checkpoint. 4. The exporter writes the batch to a durable local outbox before attempting the network request. 5. The exporter posts the batch to `POST /api/v1/ingest/session`. 6. The server stores raw records, parses them, and returns the highest contiguous acknowledged source position. 7. The exporter removes acknowledged batches and advances its local cursor. Hooks and reconciliation use the same source adapters and acknowledgement protocol. Reconciliation is a recovery path, not a separate ingestion system. ## JSONL sources JSON Lines stores one JSON value per line. Observal treats a line as ready only after its terminating newline is present. An incomplete final line remains local until the harness finishes writing it. Blank lines are ignored. The exporter retains three pieces of source position: * the zero-based record index * the absolute byte offset at the end of each record * the local byte cursor used for the next read Raw records are not normalized in the CLI. The server selects the parser registered for the harness, which keeps source discovery independent from event classification. OpenCode and Pi use native extension APIs rather than reading another process's transcript. Their extensions convert messages to stable JSONL-compatible records and then follow the same indexed ingest contract. ## Discovery and wake-ups Each harness adapter knows where that harness stores sessions and how to identify a stable session ID. Adapters may discover JSONL files, materialized hook records, or native message records. Hooks and extensions wake delivery at supported lifecycle boundaries, including prompt submission, idle, stop, and agent completion. A wake-up is only a request to resume work. Delivery state lives in the durable outbox and checkpoints, so correctness does not depend on one hook process staying alive. `observal agent pull` installs the appropriate session hooks or extension for the selected harness. `observal doctor` diagnoses missing instrumentation and can repair it with confirmation. ## Durable local delivery Session delivery is eventual at-least-once delivery with effectively-once storage. | Exporter | Durable pending data | Cursor or state | | - | - | - | | Shared Python exporter | `~/.observal/telemetry_buffer.db` | `~/.observal/sync_state.json` | | OpenCode extension | `~/.observal/opencode_session_outbox/` | Per-session state in the same directory | | Pi extension | `~/.observal/pi_session_outbox/` | `~/.observal/sync_state.json` | The exporters cap their outboxes at 256 MiB. Reaching the limit raises or logs an explicit error instead of evicting unacknowledged records. A network timeout, process exit, or server outage leaves the pending batch intact. A later hook, extension event, or reconciliation run retries it. The shared exporter drains older pending records before sending newly discovered source records. Pending data is bound to its destination server and authenticated user. It is not silently delivered to a different registry or account after a login change. ## Checkpoints and idempotency The server identifies a source record by project, user, harness, session ID, and source index. Repeated or overlapping uploads therefore converge on the same record. An acknowledgement contains: * `acknowledged_line`, the highest contiguous source index stored by the server * `acknowledged_offset`, the corresponding source byte offset Only a contiguous acknowledgement advances the local cursor. If the server has indexes 0 and 2 but not 1, the checkpoint remains at 0. This prevents a later record from hiding a missing earlier record. If local cursor state is missing, invalid, or stale, the exporter reads the authenticated checkpoint from `GET /api/v1/ingest/session/checkpoint`. It resumes from the server position when that position matches the local source. This avoids replaying all acknowledged history after local state loss. If the content at an acknowledged source index changes, the server rejects the conflict rather than overwriting the canonical record. ## Finalization and integrity repair Normal hook uploads hash only the records needed for the current batch. They do not repeatedly hash the complete session. At finalization, the exporter makes a stable-file pass and sends the total record count, total byte offset, and SHA-256 session hash. The server compares that summary with stored records. If it finds a gap or mismatch, it returns `repair_from_line`. The exporter rewinds to that point and replays the affected range through the same idempotent path. This repair handles interrupted uploads and stale cursors. It cannot recover a source record that the harness deleted before any installed hook or extension observed and durably spooled it. ## Server processing The ingest endpoint validates batch size, line size, ordered byte offsets, finalization metadata, and authentication. It stores raw source records in ClickHouse `session_events` and classifies them with the registered harness parser. Session records include agent ID, agent version, and layer hash when attribution is available. Claude Code subagent records can also include a parent session ID. Kiro can report total session credits as durable metadata. After ingestion, session-level aggregates are available through `session_stats_agg`. PostgreSQL remains responsible for relational registry, user, review, and configuration data. MCP commands and remote URLs remain direct; Observal does not proxy MCP transport traffic. ## Reconciliation Run reconciliation when hooks were installed late, a machine was offline, delivery was interrupted, or you want to verify recent local history. Preview the default seven-day scan without sending data: ```bash theme={null} observal reconcile --dry-run ``` Push recent sessions from every installed harness: ```bash theme={null} observal reconcile ``` Limit discovery to one harness or time window: ```bash theme={null} observal reconcile --harness kiro --since 24 ``` `--since` accepts 1 through 8,760 hours and defaults to 168. Without `--harness`, the command scans every installed adapter. A non-dry run validates its targets before touching the outbox, retries pending batches, recovers each server checkpoint, sends only records after that checkpoint, and finalizes fully uploaded but unfinished sessions. A dry run reports sessions whose local source is larger than the local cursor and sessions that still need final metadata. It does not drain the outbox, contact the ingest endpoint, or change cursor state. JSON results distinguish delivered, finalized, queued, rejected, skipped, and failed sources. ## Installation and verification Check authentication and local delivery health: ```bash theme={null} observal auth status observal ops telemetry status ``` Preview recoverable sessions, push them, and inspect recent traces: ```bash theme={null} observal reconcile --dry-run observal reconcile observal ops traces --limit 5 ``` If instrumentation is missing, run Doctor and confirm the proposed repair: ```bash theme={null} observal doctor ``` ## Troubleshooting ### Sessions are not discovered Confirm that the harness is installed, its local session source exists, and the source was modified inside the `--since` window. Use `--harness` to isolate one adapter. ### Records remain pending Run `observal ops telemetry status`, verify the configured server is reachable, and confirm the current login matches the user and destination associated with the pending records. The next exporter wake-up or `observal reconcile` retries them. ### A checkpoint does not match the local source The command leaves the source untouched and reports the mismatch. Preserve both the source and `~/.observal/` state while investigating. A changed or truncated transcript can no longer prove the byte position previously acknowledged by the server. ### A record is permanently rejected The shared exporter quarantines permanently rejected payloads in `~/.observal/telemetry_buffer.rejected.jsonl` so one invalid session does not block later sessions. Inspect the rejection reason and source record before retrying or removing it. ### Hooks are missing Run `observal doctor`. Doctor reports the exact metadata or harness-file changes and asks for confirmation before applying repairs. # Setup guide Source: https://docs.observal.io/getting-started/onboarding Connect your machine to Observal with two commands. It takes about two minutes and does not need Python or Docker. ## 1. Run the two commands for your OS **macOS / Linux** ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash observal auth login --server https://internal.observal.io ``` **Windows** (in PowerShell, not Command Prompt) ```powershell theme={null} irm https://raw.githubusercontent.com/Observal/Observal/main/install.ps1 | iex observal auth login --server https://internal.observal.io ``` The first command installs the `observal` CLI. The second connects it to the server. ## 2. Answer the login prompts 1. **Login method**: pick the one your team uses. If unsure, choose **Web sign-in** and approve the code in your browser. 2. **`Fix all warnings? (configures telemetry and installs AI skills for detected harnesses) [Y/n]`**: answer **Y** (or press Enter). Login installs the bundled Observal skills, and the Pi extension when Pi is detected, before this prompt. Answering **Y** also applies the supported telemetry hooks for your detected coding tools. If you answer **n**, other harnesses may still need instrumentation before sessions appear. Run `observal doctor` to check, then `observal doctor patch --all-harnesses` to install missing hooks. 3. Restart any coding tool that was open during setup so it picks up the changes. ## 3. Check it worked ```bash theme={null} observal auth whoami ``` Then open [internal.observal.io](https://internal.observal.io). Your sessions appear under Traces after your next coding session. ## Troubleshooting | Problem | Fix | | - | - | | `observal: command not found` | Open a new terminal window. On macOS/Linux, also check that `/usr/local/bin` is on your `PATH`. | | Intel Mac: download fails | There is no standalone binary for Intel Macs yet. Run `uv tool install observal-cli` instead of the first command. | | PowerShell refuses to run `irm ... \| iex` | Run `Set-ExecutionPolicy -Scope Process Bypass` in the same window, then retry. | | Login can't reach the server | Check that you're on the VPN, then run `curl https://internal.observal.io/health`. | | Answered **n** to the prompt, or added a new coding tool later | Run `observal doctor patch --all-harnesses`. | To upgrade later, run `observal self upgrade`. # Quickstart Source: https://docs.observal.io/getting-started/quickstart Go from zero to "my first trace in the Observal dashboard" in about five minutes. This assumes you have Docker running. By the end of this guide you will have: * The Observal CLI installed * An Observal server running locally * The CLI logged in as an admin * At least one MCP server instrumented * A live trace visible in the web UI ## 1. Install the CLI ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash ``` No Python required. For alternative install methods, see [Installation](/getting-started/installation). > [!NOTE] > > You need Docker Engine ≥ 24.0 with Compose v2 (`docker compose`, not `docker-compose`). Homebrew's Docker formula is outdated. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages. Verify with `docker version` and `docker compose version`. ## 2. Start the server ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal cp .env.example .env docker compose -f docker/docker-compose.yml up --build -d ``` That's it. The `.env.example` ships with working defaults. The core services come up: | Service | URL | Purpose | | - | - | - | | `observal-lb` (nginx) | `http://localhost` | Reverse proxy (API + Web) | | `observal-web` | `http://localhost:3000` | Web UI (Vite, direct) | | `observal-api` | internal | FastAPI backend | | `observal-worker` | internal | Background jobs (arq) | | `observal-init` | internal | Runs DB migrations, then exits | | `observal-db` | `localhost:5432` | PostgreSQL 18 | | `observal-clickhouse` | `localhost:8123` | ClickHouse | | `observal-redis` | `localhost:6379` | Redis | Optional monitoring can be enabled with `make up-prometheus` or `make up-observability`. Prometheus listens on `http://localhost:9090`; Grafana listens on `http://localhost:3001` when the Grafana profile is enabled. The API waits for Postgres, ClickHouse, and Redis to pass health checks before starting. Expect 15–30 seconds. Confirm it is up: ```bash theme={null} curl http://localhost/health # → {"status": "ok"} ``` Hitting a port conflict? See [Self-Hosting → Ports and volumes](/self-hosting/ports-and-volumes). ## 3. Log in ```bash theme={null} observal auth login ``` Prompts: 1. **Server URL**: press Enter for `http://localhost` 2. **Login method**: pick `[E]mail` 3. **Email / password**: use one of the seeded demo accounts: | Role | Email | Password | | - | - | - | | Super Admin | `super@demo.example` | `super-changeme` | | Admin | `admin@demo.example` | `admin-changeme` | | Reviewer | `reviewer@demo.example` | `reviewer-changeme` | | User | `user@demo.example` | `user-changeme` | Log in as super admin for the fewest restrictions while exploring. Credentials land in `~/.observal/config.json` (mode `0600`). Check it worked: ```bash theme={null} observal auth whoami # → super@demo.example (super_admin) ``` ## 4. Discover and instrument your harness If you already have MCP servers configured in Claude Code, Kiro, Cursor, VS Code, or Copilot, first see what's there: ```bash theme={null} observal scan ``` Expected output lists detected harnesses, MCP servers, skills, hooks, and agents. MCP commands and remote URLs are shown exactly as configured. `scan` is read-only: it shows what you have without modifying anything. Install session telemetry hooks: ```bash theme={null} observal doctor patch --all-harnesses ``` `doctor patch` installs supported session hooks and extensions. It does not rewrite MCP configuration. Restart your harness so the hook changes take effect, then begin a coding session. ## 5. See your first trace Open `http://localhost/traces` in your browser. Start a prompt in your harness and let the session complete. Refresh to see the indexed session and its parsed events. Or use the CLI: ```bash theme={null} observal ops traces --limit 5 ``` ## 6. (Optional) Pull an agent Browse what the community has published: ```bash theme={null} observal agent list observal agent show / ``` Install one into your harness: ```bash theme={null} observal agent pull / --harness ``` This writes agent files, skills, hooks, and direct MCP configs into the right places for your harness. ## What you just built ```mermaid theme={null} flowchart LR harness[Your harness] transcript[Local session transcript] hook[Session hook or extension] api[Session ingest API] ch[(ClickHouse)] ui[Web UI] harness --> transcript hook --> transcript hook --> api api --> ch ch --> ui ``` Observal indexes session transcript records into canonical events and aggregates. ## Where to next | You want to... | Go to | | - | - | | Understand registry identity | [Core concepts](/core-concepts) | | Understand session tracking | [Session tracking and reconciliation](/core-concepts/session-tracking) | | Learn what to do with traces | [Use Cases](/use-cases) | | Configure the server for production | [Self-Hosting](/self-hosting) | | Deep-dive on a CLI command | [CLI Reference](/cli) | # Installation Source: https://docs.observal.io/index Observal has two parts: a **server** you self-host and a **CLI** installed on each developer machine. ## Install the server The server runs as a Docker Compose stack (API, web UI, PostgreSQL, ClickHouse, Redis, worker, load balancer). Prometheus and Grafana are optional deployment overlays. > [!NOTE] > > Requires Docker Engine ≥ 24.0 with Compose v2 (`docker compose`, not `docker-compose`). Homebrew's Docker formula is outdated. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages. Verify with `docker version` and `docker compose version`. **One-line install:** ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install-server.sh | bash ``` This downloads a config package, runs guided setup (domain, secrets, ports), pulls container images from GHCR, and starts the full stack. **From source** (for contributors): ```bash theme={null} git clone https://github.com/Observal/Observal.git && cd Observal cp .env.example .env make up ``` For deployment options, see [Self-Hosting](/self-hosting/docker-compose) and [Production deployment](/self-hosting/production-deploy). ## Install the CLI The CLI is what you use to log in, instrument harness configs, pull agents, and query traces. ## Install (standalone binary) The standalone binary is the simplest way to install. No Python required. ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash ``` This downloads the latest release binary for your platform and places it on your `PATH`. The installer downloads the release binary and checks its SHA-256 hash against the release checksums when available. If it cannot download the checksums, it warns and proceeds without verification; it does not create a key in `~/.observal/config.json`. Verify it worked: ```bash theme={null} observal --version ``` ## Alternative: install with Python If you prefer to install via Python, use one of these methods. Requires Python 3.11 or newer. **uv (recommended):** ```bash theme={null} uv tool install observal-cli ``` **pipx:** ```bash theme={null} pipx install observal-cli ``` **pip:** ```bash theme={null} pip install --user observal-cli ``` ### Optional extras The Python package includes the Docker SDK. The `migrate` extra adds `pyarrow` for `observal server migrate`: ```bash theme={null} uv tool install 'observal-cli[migrate]' ``` ## Install from source (for contributors) ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal uv tool install --editable . ``` ## What gets installed The Python package provides three entry points: | Command | Purpose | | - | - | | `observal` | The main CLI | | `observal-sandbox-run` | Sandbox runner invoked by Observal sandboxes | | `observal-sandbox-mcp` | MCP adapter used by sandbox execution | You will almost never call the sandbox helpers directly. The CLI wires them into your harness config for you. ## Upgrade ```bash theme={null} observal self upgrade ``` ## Uninstall Standalone binary: ```bash theme={null} rm "$(which observal)" ``` Python install: ```bash theme={null} uv tool uninstall observal-cli # or: pipx uninstall observal-cli # or: pip uninstall observal-cli ``` Uninstalling the CLI does **not** remove your config (`~/.observal/`). Delete that folder if you want a clean slate: ```bash theme={null} rm -rf ~/.observal ``` ## Next -> [Quickstart](/getting-started/quickstart) # Copilot Source: https://docs.observal.io/integrations/copilot Copilot is a first-class Observal harness integration. Observal can install GitHub Copilot agents, configure MCP servers, expose skills, and bridge telemetry hooks. *** ## Overview Copilot agent profiles are Markdown files stored in `.github/agents/`. Copilot currently supports project installation scope only. MCP servers, hooks, skills, and agent profiles are managed through the project configuration. *** ## Supported capabilities | Capability | Support | | - | - | | Agent profiles | Project scope | | Hook bridge | Supported | | MCP servers | `.vscode/mcp.json` | | Skills | `.github/skills/{name}/SKILL.md` | | Session parsing | Built-in `copilot-cli` session parser | | Default scope | Project | *** ## Setup ### 1. Install the Observal CLI ```bash theme={null} uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash theme={null} observal auth login ``` This writes credentials to `~/.observal/config.json`. ### 3. Pull an agent into Copilot ```bash theme={null} observal agent pull --harness copilot ``` By default, the agent is written to: ``` .github/agents/{name}.agent.md ``` *** ## Config paths | Purpose | Project scope | | - | - | | Agent profile | `.github/agents/{name}.agent.md` | | MCP config | `.vscode/mcp.json` | | Skills | `.github/skills/{name}/SKILL.md` | | Hook config | `.github/hooks/{name}.json` | Copilot MCP configuration uses the `servers` key. *** ## Hook spec ### Event map | Observal event | Copilot event | | - | - | | `SessionStart` | `SessionStart` | | `UserPromptSubmit` | `UserPromptSubmit` | | `PreToolUse` | `PreToolUse` | | `PostToolUse` | `PostToolUse` | | `Stop` | `Stop` | *** ## Session parsing and delivery Copilot uses the built-in `copilot-cli` server parser while retaining separate source adapters for VS Code Copilot and Copilot CLI. VS Code hook events are durably materialized as JSONL-compatible source records; Copilot CLI discovers its native `events.jsonl` sessions. Both adapters route through the shared spool-first delivery engine. Hook execution writes to the durable outbox and defers network work, retries keep stable source indexes, and cursors advance only after contiguous server acknowledgement. Recovery uses the authenticated server checkpoint. Finalization audits the complete source hash and rewinds/replays a mismatched range. *** ## Caveats **Only project scope is supported.** Agents are installed into `.github/agents/`. **Configuration is split across project directories.** MCP servers are stored in `.vscode/mcp.json`, while hooks and skills live under `.github/`. **Rules are not supported.** Copilot supports agent profiles, MCP servers, skills, and hooks, but does not expose a rules configuration. # Cursor Source: https://docs.observal.io/integrations/cursor Cursor is a first-class Observal harness integration. Observal can install Cursor agents, configure MCP servers, and collect Cursor session telemetry. *** ## Overview Cursor agent profiles are Markdown files. Project agents live in `.cursor/agents/`. User agents live in `~/.cursor/agents/`. Cursor supports project and user installation scopes. MCP servers, hooks, and agent profiles are managed under the `.cursor` configuration directory. *** ## Supported capabilities | Capability | Support | | - | - | | Agent profiles | Project and user scope | | Hook bridge | Supported | | MCP servers | `.cursor/mcp.json` and `~/.cursor/mcp.json` | | Session parsing | Built-in Cursor session parser | | Default scope | Project | *** ## Setup ### 1. Install the Observal CLI ```bash theme={null} uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash theme={null} observal auth login ``` This writes credentials to `~/.observal/config.json`. ### 3. Pull an agent into Cursor ```bash theme={null} observal agent pull --harness cursor ``` Cursor's default scope is project scope. By default, the agent is written to `.cursor/agents/{name}.md`. To install into your user configuration: ```bash theme={null} observal agent pull --harness cursor --scope user ``` User agents are written to `~/.cursor/agents/{name}.md`. *** ## Config paths | Purpose | Project scope | User scope | | - | - | - | | Agent profile | `.cursor/agents/{name}.md` | `~/.cursor/agents/{name}.md` | | MCP config | `.cursor/mcp.json` | `~/.cursor/mcp.json` | | Skills | `.cursor/skills/{name}/SKILL.md` | `~/.cursor/skills/{name}/SKILL.md` | | Hook config | `.cursor/hooks.json` | `~/.cursor/hooks.json` | Cursor MCP configs use the `mcpServers` key. *** ## Hook spec ### Event map | Observal event | Cursor event | | - | - | | `PreToolUse` | `preToolUse` | | `PostToolUse` | `postToolUse` | | `Stop` | `sessionEnd` | | `SessionStart` | `sessionStart` | | `UserPromptSubmit` | `beforeSubmitPrompt` | | `SubagentStop` | `subagentStop` | *** ## Session delivery and parsing Cursor keeps its built-in `cursor` parser. Its adapter resolves the transcript path supplied by Cursor, with project-directory discovery as a fallback, and discovers separately stored subagent transcripts. Cursor hook timeouts require a split delivery path: the hook synchronously writes complete transcript records and Stop-event token usage into the durable SQLite outbox, then a detached shared worker performs network delivery. The local byte/line cursor is not advanced by the hook or worker launch; it advances only after the server acknowledges a contiguous checkpoint. A delayed finalizer captures records written after Cursor's Stop event, restores stale local state from the server checkpoint, and performs the final hash audit. Audit mismatches rewind and replay the affected range. *** ## Caveats **Default scope is project.** `observal agent pull --harness cursor` writes to `.cursor/agents/` unless `--scope user` is specified. **Configuration lives under `.cursor`.** Agent profiles, MCP configuration, skills, and hooks are stored in the `.cursor` directory for project installs and under `~/.cursor` for user installs. **Auto model sentinel configuration is not available.** # Goose Source: https://docs.observal.io/integrations/goose [Goose](https://goose-docs.ai/) is a first-class Observal harness integration. Observal can install Goose custom agents, configure MCP extensions, expose skills, install a Goose hook plugin, and collect Goose session telemetry. Supported Goose versions: **1.10.0 and later** (the release that moved session storage to SQLite), verified against **goose 1.45.0**. Goose CLI and Goose Desktop share the same configuration, skills, agents, plugins, and session database, so a single install covers both. *** ## Overview Goose custom agents are Markdown files with YAML frontmatter. Project agents live in `.agents/agents/`, user agents in `~/.agents/agents/`. MCP servers are Goose *extensions*. Goose reads them from a single user-level `config.yaml`, so Observal merges its extensions into that file and leaves every other key — providers, models, global settings — untouched. Session telemetry uses a Goose hook plugin. `observal doctor patch --harness goose` writes `~/.agents/plugins/observal/`, containing a `plugin.json` manifest and a `hooks/hooks.json` file. The hooks run `observal_cli.hooks.session_push --harness goose`, which reads Goose's SQLite session store and pushes new records through Observal's shared acknowledged delivery pipeline. *** ## Supported capabilities | Capability | Support | | - | - | | Agent profiles | Project and user scope | | Hook bridge | Goose plugin (`hooks/hooks.json`) | | Hook events | `SessionStart`, `UserPromptSubmit`, `Stop`, `SessionEnd` | | MCP servers | `extensions` in `~/.config/goose/config.yaml` | | MCP transports | `stdio` and `streamable_http` | | Agent prompt | Registry prompts are written into the generated Goose agent file | | Skills | `.agents/skills/{name}/SKILL.md` and `~/.agents/skills/{name}/SKILL.md` | | Session parsing | SQLite session store mirrored to JSONL | | Telemetry | Hook push plus `observal reconcile` | | Model selection | Registry-backed Goose model catalog | *** ## Setup ### 1. Install the Observal CLI ```bash theme={null} uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash theme={null} observal auth login ``` ### 3. Pull an agent into Goose ```bash theme={null} observal agent pull --harness goose ``` Goose's default scope is user scope, so the agent is written to `~/.agents/agents/{name}.md`. To install into the current project: ```bash theme={null} observal agent pull --harness goose --scope project ``` MCP extensions are always written to `~/.config/goose/config.yaml`, because that is the only file Goose reads extensions from. ### 4. Install or refresh the Goose hook plugin ```bash theme={null} observal doctor patch --harness goose ``` Restart your Goose session afterwards so the plugin is discovered. ### 5. Check what is installed ```bash theme={null} observal scan --harness goose observal doctor ``` *** ## Config paths | Purpose | Project scope | User scope | | - | - | - | | Agent profile | `.agents/agents/{name}.md` | `~/.agents/agents/{name}.md` | | MCP extensions | — | `~/.config/goose/config.yaml` | | Skill definition | `.agents/skills/{name}/SKILL.md` | `~/.agents/skills/{name}/SKILL.md` | | Hook plugin | `.agents/plugins/observal/` | `~/.agents/plugins/observal/` | | Session database | — | `~/.local/share/goose/sessions/sessions.db` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Observal lockfile | `~/.observal/lockfile.json` | `~/.observal/lockfile.json` | User-scope locations follow Goose's own resolution order: 1. `$GOOSE_PATH_ROOT` when set to an absolute path — config becomes `$GOOSE_PATH_ROOT/config`, data `$GOOSE_PATH_ROOT/data`, and skills, agents and plugins move to `$GOOSE_PATH_ROOT/.agents/`. 2. `$XDG_CONFIG_HOME` / `$XDG_DATA_HOME` when set. 3. `~/.config/goose` and `~/.local/share/goose`. On Windows, Goose reads `%APPDATA%\Block\goose\config\config.yaml` and `%APPDATA%\Block\goose\data\sessions\sessions.db`. Scanning, `doctor`, and session reading resolve those locations automatically. `observal agent pull` still writes the documented Unix paths, so Windows users should move the generated `config.yaml` extensions into the `%APPDATA%` config by hand. *** ## Agent profile format ```markdown theme={null} --- name: my-agent description: Reviews code for correctness and risk model: gpt-5.5 --- You are a Goose agent with the following specialization... ``` Only `name` is required by Goose. `model` is emitted when the install resolves a model from the Goose model catalog. *** ## Extension format Observal writes MCP servers as Goose extensions: ```yaml theme={null} extensions: filesystem: type: stdio name: filesystem enabled: true cmd: npx args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] envs: {} env_keys: [] timeout: 300 remote-tools: type: streamable_http name: remote-tools enabled: true uri: https://example.com/mcp headers: {} envs: {} env_keys: [] timeout: 300 ``` Observal writes MCP servers as `stdio` or `streamable_http` extensions and only ever touches the entries it owns. `observal scan` additionally reports legacy `sse` extensions. Goose's `builtin`, `platform` and `frontend` extensions run in-process and `inline_python` carries code rather than a command, so none of them are reported or rewritten. *** ## Hook plugin spec `~/.agents/plugins/observal/hooks/hooks.json`: ```json theme={null} { "hooks": { "SessionStart": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }], "UserPromptSubmit": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }], "Stop": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }], "SessionEnd": [{ "hooks": [{ "type": "command", "command": "...", "timeout": 30 }] }] } } ``` Rules deliberately omit `matcher`: Goose treats `matcher` as a regular expression, so a bare `"*"` would be invalid and the rule would be skipped. Goose runs every hook through `sh -c` on all platforms, passes the event JSON on stdin, waits for the command, and enforces the rule's `timeout`. The Observal hook therefore only reads Goose's database and appends to the local outbox; the network upload is handed to a detached worker so a slow or unreachable Observal server can never stall a Goose turn. The hook writes nothing to stdout and never exits `2`, so it can never block a `PreToolUse` tool call or a `Stop` turn boundary. `doctor patch` preserves any non-Observal rules you add to the same plugin. Goose records each discovered plugin under a `plugins` map in `config.yaml` keyed by its absolute path, so the plugin activates itself on the next session. To disable the plugin without removing it, add it to `disabledPlugins` in `~/.config/goose/settings.json`. To remove it entirely: ```bash theme={null} observal doctor cleanup --harness goose ``` ### Hook payload coverage Verified against goose 1.45.0: | Event | `working_dir` | Notes | | - | - | - | | `SessionStart` | no | Fires once per session | | `UserPromptSubmit` | no | Carries the prompt in `message` | | `Stop` | yes | Fires at the end of **every turn**, not the session | | `SessionEnd` | no | Emitted by the goose **CLI** only | Because most events omit `working_dir`, Observal reads the session's working directory from `sessions.working_dir` instead, which keeps agent attribution and layer hashing correct on every event. *** ## Session push behavior Goose stores sessions in SQLite rather than JSONL, so the CLI projects each session onto an append-only JSONL mirror at `~/.observal/sessions/goose/.jsonl`. Everything downstream — the durable outbox, acknowledged delivery, checkpoint recovery, and the final SHA-256 audit — is the shared engine used by every other harness. 1. A hook event resolves the Goose session id from the payload. 2. `sessions.db` is opened through a `file:...?mode=ro` URI with a busy timeout, so Observal never writes to, migrates, or checkpoints Goose's database, and never blocks a Goose session that is writing concurrently. 3. Rows newer than the mirror's cursor are appended as JSONL records. The mirror only grows, which keeps previously acknowledged byte offsets valid. 4. The batch is spooled to the local outbox and a detached worker performs the upload, so the hook returns in milliseconds regardless of server health. 5. `SessionEnd` appends a `session_end` boundary carrying the session's token and cost totals, then finalizes delivery with an integrity hash. 6. Delegated (subagent) sessions linked by `parent_session_id` are mirrored and delivered alongside their parent. Mirror records: | Record | Meaning | | - | - | | `{"type": "session", ...}` | Session metadata: name, working directory, provider, model, parent session | | `{"type": "message", ...}` | One `messages` row: role, content blocks, per-message token usage | | `{"type": "session_end", ...}` | Final boundary with accumulated token and cost totals | Goose content blocks are parsed into trace events: `text`, `thinking`, `redactedThinking`, `toolRequest`, `toolResponse` (including failures), `toolConfirmationRequest`, and `error`. ### Backfilling existing sessions ```bash theme={null} observal reconcile --harness goose observal reconcile --harness goose --since 720 --dry-run ``` Reconcile reads the same read-only database, mirrors any session updated within the window, and drains it through the outbox. *** ## Caveats **The hook plugin is shared per machine.** It is installed by `doctor patch`, not by each agent pull. **MCP extensions are user scope only.** Goose reads extensions from one `config.yaml`, so `--scope project` still writes extensions to the user config. Agents and skills honour the requested scope. Goose rewrites that same file when it registers plugins, and neither Goose nor Observal preserves YAML comments in it. **Goose Desktop sessions finalize later.** Only the goose CLI emits `SessionEnd`. Desktop sessions are still captured turn by turn through `Stop`; their final integrity audit runs from background recovery or `observal reconcile`. **Legacy JSONL sessions are not imported.** Goose imports pre-1.10 `.jsonl` files into `sessions.db` on upgrade, and Observal reads the database, so those sessions are covered. Leftover `.jsonl` files on disk are ignored. **Rewritten conversations replay.** `goose session --resume --edit` rewrites a session's messages. The mirror is append-only, so the rewritten history is appended rather than replacing what was already delivered. **Late writes land on the next event.** If Goose persists a message after the `SessionEnd` hook has read the database, that row is delivered by the next hook event or by `observal reconcile`. # Kiro Source: https://docs.observal.io/integrations/kiro Kiro is a first-class Observal harness integration. Observal can install Kiro agents, configure MCP servers, add hooks, expose skills, and collect Kiro session telemetry. *** ## Overview Kiro agent profiles are JSON files. Project agents live in `.kiro/agents/`. User agents live in `~/.kiro/agents/`. When Observal installs a Kiro agent, it writes telemetry hooks into a standalone hooks file — `~/.kiro/hooks/observal.json` (user scope) or `.kiro/hooks/observal.json` (project scope) — in the v1 schema, using the `UserPromptSubmit` and `Stop` triggers. Both hooks run the shared `observal_cli.hooks.session_push --harness kiro` entry point. Kiro IDE 1.0 and CLI 3.0 read the standalone file; the IDE does not fire inline agent hooks. Once installed, shared hooks collect **all** Kiro sessions in that scope, including ordinary conversations using MCPs or skills. Agent attribution is separate: sessions without a verified registry agent remain unattributed. Inline hooks do **not** hide an agent from the IDE agent picker. What does is `allowedTools` or `toolsSettings` without a `permissions` block: Kiro IDE 1.x `ProfileLoader` rejects such a profile outright (`reasonCode cli_only_agent`). Observal no longer emits either field and strips them on pull. ### Legacy Kiro CLI 2.x Kiro CLI 2.x only understands inline agent hooks. When Observal detects CLI 2.x, it additionally writes legacy inline `userPromptSubmit`/`stop` hooks into the agent JSON, even when the IDE is installed. Ordinary CLI 2.x sessions do not fire these agent-specific hooks; use `observal reconcile --harness kiro` to backfill them. Detection can be overridden with `OBSERVAL_KIRO_CLI_VERSION` and `OBSERVAL_KIRO_IDE` (`1`/`0`). The hook reads Kiro CLI JSONL files from `~/.kiro/sessions/cli/` and IDE transcripts from workspace session directories. It sends new records since the last push. Reconciliation backfills missed CLI and IDE sessions, including separately stored delegated sub-executions. *** ## Supported capabilities | Capability | Support | | - | - | | Agent profiles | Project and user scope | | Hook bridge | `UserPromptSubmit` and `Stop` in `.kiro/hooks/observal.json` | | Custom hooks | `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `Stop`, `PostFileSave`, `PostFileCreate`, `PostFileDelete` | | Legacy CLI 2.x | Inline agent hooks, only when no Kiro IDE is installed | | MCP servers | `.kiro/settings/mcp.json` and `~/.kiro/settings/mcp.json` | | Agent prompt | Registry prompts are embedded in the generated Kiro agent profile | | Guidance files | Scanned from steering files and `AGENTS.md`, not overwritten | | Skills | `.kiro/skills/{name}/SKILL.md` and `~/.kiro/skills/{name}/SKILL.md` | | Session parsing | Kiro JSONL parser | | Telemetry | Kiro session transcripts delivered through hooks and reconciliation | | Model selection | Registry-backed Kiro model catalog | *** ## Setup ### 1. Install the Observal CLI ```bash theme={null} uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash theme={null} observal auth login ``` This writes credentials to `~/.observal/config.json`. ### 3. Pull an agent into Kiro ```bash theme={null} observal agent pull --harness kiro ``` Kiro's default scope is user scope. By default, the agent is written to `~/.kiro/agents/{name}.json`. To install into the current project: ```bash theme={null} observal agent pull --harness kiro --scope project ``` Project agents are written to `.kiro/agents/{name}.json`. ### 4. Refresh Kiro hooks Pull the agent again to refresh its Observal hook commands. `doctor patch` does not install generic Kiro hooks: pulling an agent installs shared hooks for its scope. These hooks do not carry that agent's UUID and capture ordinary sessions too. CLI 2.x still needs per-agent inline hooks. *** ## Config paths | Purpose | Project scope | User scope | | - | - | - | | Agent profile | `.kiro/agents/{name}.json` | `~/.kiro/agents/{name}.json` | | Guidance files | `.kiro/steering/*.md`, `AGENTS.md` | `~/.kiro/steering/*.md` | | MCP config | `.kiro/settings/mcp.json` | `~/.kiro/settings/mcp.json` | | Skill definition | `.kiro/skills/{name}/SKILL.md` | `~/.kiro/skills/{name}/SKILL.md` | | Hook config | Embedded in `.kiro/agents/{name}.json` | Embedded in `~/.kiro/agents/{name}.json` | | Custom hook scripts | `.kiro/hooks/` | `~/.kiro/hooks/` | | Session JSONL | `~/.kiro/sessions/cli/{session_id}.jsonl` | `~/.kiro/sessions/cli/{session_id}.jsonl` | | Credit metadata | `~/.kiro/sessions/cli/{session_id}.json` | `~/.kiro/sessions/cli/{session_id}.json` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Last session cache | `~/.observal/.kiro-session` | `~/.observal/.kiro-session` | Kiro MCP configs use the `mcpServers` key. *** ## Hook spec Observal writes the telemetry hooks to the standalone v1 hooks file — `~/.kiro/hooks/observal.json` (user scope) or `.kiro/hooks/observal.json` (project scope): ```json theme={null} { "version": "v1", "hooks": [ { "name": "observal-session-push-userpromptsubmit", "trigger": "UserPromptSubmit", "action": { "type": "command", "command": "python -m observal_cli.hooks.session_push --harness kiro" } }, { "name": "observal-session-push-stop", "trigger": "Stop", "action": { "type": "command", "command": "python -m observal_cli.hooks.session_push --harness kiro" } } ] } ``` The command carries no `OBSERVAL_AGENT_ID`. One file per scope serves every agent, so an id baked into it would attribute every session to whichever agent was pulled last. Attribution comes from session metadata instead. On non-Windows platforms, generated server config may use `python3` instead of `python`. During `observal agent pull`, the CLI rewrites Observal hook commands to use the active Python interpreter. ### Legacy CLI 2.x inline hooks Kiro CLI 2.x predates the standalone file and reads hooks only from the agent JSON. On a machine whose `kiro-cli` reports major version 2, Observal *also* writes inline `userPromptSubmit`/`stop` hooks into each agent profile, carrying `OBSERVAL_AGENT_ID` since an agent profile belongs to exactly one agent: ```json theme={null} { "hooks": { "userPromptSubmit": [ { "command": "OBSERVAL_AGENT_ID= python -m observal_cli.hooks.session_push --harness kiro" } ] } } ``` Both formats are written on such machines. Having the IDE installed is not a reason to withhold the inline copy: the two surfaces coexist, and the IDE loads an agent carrying inline hooks and simply never fires them. ### Attribution The hooks file is shared by every agent, so the hook command cannot identify one. Attribution is resolved from the session itself: 1. A session started as an agent — picked from the IDE's agent dropdown — records that agent in `session_start.agentType`, and the agent's own profile hooks fire with an agent-scoped `hookId`. Either names the agent. 2. A session that delegates records `sub_agent_start` with `subAgentName`, and the delegated agent's work is written to a separate child transcript. The parent conversation is collected but is **not** attributed to the delegate. 3. CLI sessions carry the active agent in the session metadata written beside the transcript. 4. The CLI selects the active server URL under `registries` in `~/.observal/lockfile.json`, then looks up that name under that registry's `kiro` harness. 5. A verified agent receives its lockfile id and version. Ordinary sessions and unknown agents are still collected, with no agent id rather than a guessed one. Legacy CLI 2.x inline hooks may carry `OBSERVAL_AGENT_ID`. ### Event map | Observal event | Kiro event | | - | - | | `SessionStart` | `agentSpawn` | | `UserPromptSubmit` | `userPromptSubmit` | | `PreToolUse` | `preToolUse` | | `PostToolUse` | `postToolUse` | | `Stop` | `stop` | `preToolUse` and `postToolUse` hooks can include a `matcher`. Observal uses `*` when no matcher is set. *** ## Session push behavior Kiro uses the shared acknowledged session delivery engine: 1. Resolve the Kiro session ID from the hook payload or `~/.observal/.kiro-session`. 2. Find `~/.kiro/sessions/cli/{session_id}.jsonl` through the Kiro adapter. 3. Read complete records after the acknowledged byte/line cursor. 4. Persist new batches to `~/.observal/telemetry_buffer.db` before network delivery. 5. Retry batches idempotently until the server returns a contiguous checkpoint covering them. 6. Advance the local cursor only to that acknowledged checkpoint. 7. Recover missing or corrupt local state from the authenticated server checkpoint. 8. On finalization, compare the SHA-256 audit manifest and replay any affected range. On `stop`, a delayed stable-file pass captures late records and finalizes the cursor. The adapter also reads `~/.kiro/sessions/cli/{session_id}.json` and durably sends Kiro credit usage, including when no transcript lines were added by the final hook. *** ## Agent profile format Observal generates Markdown agent profiles like this: ```markdown theme={null} --- name: my-agent model: claude-sonnet-4 --- You are a Kiro agent with the following specialization... ``` The `model` field is present when a model is resolved for the agent. *** ## Skill file format Kiro skills live at: | Scope | Path | | - | - | | Project | `.kiro/skills/{name}/SKILL.md` | | User | `~/.kiro/skills/{name}/SKILL.md` | Example: ```markdown theme={null} --- description: "Runs the project test suite" task_type: testing --- # Run Tests Run `pytest -q` from the project root. ``` *** ## Caveats **Guidance files are scan-only.** Observal layers Kiro steering files and `AGENTS.md` as context, but does not overwrite them during pull. **Hooks are per scope, not per agent.** Pulling an agent refreshes the shared `observal.json` hooks file for that scope, which then serves every Kiro agent there — so the file exists even when no agent is locked. Pull the agent again to replace an older Kiro-specific push command with the shared acknowledged exporter. On legacy CLI 2.x, inline per-agent hooks carrying `OBSERVAL_AGENT_ID` are written in addition. **Default scope is user.** `observal agent pull --harness kiro` writes to `~/.kiro/agents/` unless `--scope project` is set. **No Claude Code subagent layout.** Kiro reads `~/.kiro/sessions/cli/{session_id}.jsonl`. It does not scan Claude Code's `subagents/` directory. **MCP config is Kiro-specific.** Kiro uses `.kiro/settings/mcp.json` and `~/.kiro/settings/mcp.json`, not Claude Code MCP paths. # OpenCode Source: https://docs.observal.io/integrations/opencode OpenCode is a first-class Observal harness integration. Observal can install OpenCode agents, configure MCP servers, expose skills, install hook plugins, and collect OpenCode session telemetry. *** ## Overview OpenCode agent profiles are Markdown files. Project agents live in `.opencode/agents/`. User agents live in `~/.config/opencode/agents/`. OpenCode session telemetry uses an in-process TypeScript plugin. The plugin is installed by `observal auth login` or `observal doctor patch`. Agent pulls write agent profiles and update the Observal lockfile, but they do not embed the telemetry plugin in each agent file. The plugin listens to OpenCode session and message events, reads message data through the OpenCode client API, converts new messages into Observal session lines, and sends them to Observal. *** ## Supported capabilities | Capability | Support | | - | - | | Agent profiles | Project and user scope | | Hook bridge | OpenCode plugin | | Plugin events | `session.created`, `session.idle`, `message.updated` | | MCP servers | `opencode.json` and `~/.config/opencode/opencode.json` | | Agent prompt | Registry prompts are written into the generated OpenCode agent profile | | Skills | `.opencode/skills/{name}/SKILL.md` and `~/.config/opencode/skills/{name}/SKILL.md` | | Session parsing | OpenCode plugin exports JSONL-compatible session lines | | Telemetry | OpenCode session transcripts delivered through the plugin and reconciliation | | Model selection | Registry-backed OpenCode model catalog | *** ## Setup ### 1. Install the Observal CLI ```bash theme={null} uv tool install observal-cli # or: pipx install observal-cli ``` ### 2. Authenticate ```bash theme={null} observal auth login ``` This writes credentials to `~/.observal/config.json`. If OpenCode is detected, login can install the Observal plugin. ### 3. Pull an agent into OpenCode ```bash theme={null} observal agent pull --harness opencode ``` OpenCode's default scope is user scope. By default, the agent is written to `~/.config/opencode/agents/{name}.md`. To install into the current project: ```bash theme={null} observal agent pull --harness opencode --scope project ``` Project agents are written to `.opencode/agents/{name}.md`. ### 4. Install or refresh the OpenCode plugin ```bash theme={null} observal doctor patch --harness opencode ``` This installs or updates `observal-plugin.ts` when the plugin is missing, stale, or different from the bundled source. *** ## Config paths | Purpose | Project scope | User scope | | - | - | - | | Agent profile | `.opencode/agents/{name}.md` | `~/.config/opencode/agents/{name}.md` | | MCP config | `opencode.json` | `~/.config/opencode/opencode.json` | | Skill definition | `.opencode/skills/{name}/SKILL.md` | `~/.config/opencode/skills/{name}/SKILL.md` | | Plugin hooks | `.opencode/plugins/{name}.ts` | `~/.config/opencode/plugins/{name}.ts` | | Observal plugin | `.opencode/plugins/observal-plugin.ts` | `~/.config/opencode/plugins/observal-plugin.ts` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Observal lockfile | `~/.observal/lockfile.json` | `~/.observal/lockfile.json` | OpenCode MCP configs use the `mcp` key. *** ## Plugin spec Observal installs a TypeScript plugin named `observal-plugin.ts`. The plugin subscribes to OpenCode runtime events: | OpenCode event | Observal use | | - | - | | `session.created` | Capture the active OpenCode agent name for the session | | `message.updated` | Mark the session as having new data to push | | `session.idle` | Fetch new messages and send them to Observal | | `session.updated` | Refresh active agent tracking when emitted by OpenCode | | `session.next.agent.switched` | Refresh active agent tracking when the active agent changes | The bundled spec version is checked by `doctor patch`, along with the plugin content hash, so stale local copies are replaced. *** ## Attribution OpenCode exposes the active agent name in session events, not an Observal UUID. The Observal lockfile maps that OpenCode agent name back to the registry agent id and installed version. 1. `observal agent pull` writes the OpenCode agent profile and records the agent name, id, version, scope, and project directory in `~/.observal/lockfile.json`. 2. The OpenCode plugin receives `session.created` or agent switch events and reads the active OpenCode agent name. 3. Built-in OpenCode agents such as `build`, `plan`, `general`, `explore`, `scout`, `compaction`, `title`, and `summary` are ignored. 4. The plugin selects the configured server URL under `registries` in `~/.observal/lockfile.json`, then searches that registry's `opencode` harness for an entry whose `name` or `id` matches the active agent. 5. If multiple entries match, a project-scoped entry for the current directory wins, then a user-scoped entry, then the first match. 6. The session payload is sent with the resolved `agent_id` and `agent_version`. 7. If no lockfile match exists, the plugin skips attribution for that session instead of guessing. *** ## Session push behavior The OpenCode plugin implements the same acknowledged delivery contract as Python harnesses: 1. Load `~/.observal/config.json` and resolve the active agent through `~/.observal/lockfile.json`. 2. On `session.idle`, recover missing, stale, or quarantined local state from the server's contiguous checkpoint. 3. Retry any durable pending batch before reading new messages. 4. Fetch messages from `client.session.messages` and convert each message into one stable JSONL-compatible source record. 5. Atomically persist the pending indexed batch under `~/.observal/opencode_session_outbox/` before network delivery. 6. Retry the same batch idempotently until the server returns a contiguous acknowledgement covering it. 7. Advance the persisted source line only after that acknowledgement; a crash after server commit safely retries the batch. 8. On finalization, compare a SHA-256 audit manifest and replay from the server-requested repair range when needed. Acknowledged state is retained for seven days to avoid unnecessary replay. Capacity errors fail closed without discarding pending records. Corrupt state is quarantined and rebuilt from the authenticated server checkpoint plus the OpenCode message source. *** ## Agent profile format OpenCode agents are Markdown files. Observal writes the agent prompt into the profile selected by scope: ```markdown theme={null} --- name: my-agent --- You are an OpenCode agent with the following specialization... ``` *** ## Caveats **The plugin is shared per OpenCode install.** It is installed by auth login or `doctor patch`, not by each agent pull. **Attribution depends on the lockfile.** If an agent profile is copied by hand without running `observal agent pull`, the plugin may see the OpenCode agent name but have no Observal id or version to send. **Built-in OpenCode agents are ignored.** Sessions for built-in agents are not attributed to registry agents. **MCP config is OpenCode-specific.** OpenCode uses `opencode.json` and the `mcp` key, not Kiro or Claude Code MCP paths. # Pi Source: https://docs.observal.io/integrations/pi Pi is a first-class Observal harness integration. Observal can install Pi agent profiles, configure MCP servers, expose skills, install the telemetry extension, and collect Pi session telemetry. *** ## Overview Pi is harness-centric: the whole of Pi is one agent. There is no per-agent profile file the way Kiro or OpenCode have one. Instead, Observal writes the agent's rules into `AGENTS.md`, which becomes Pi's system prompt, and writes MCP servers and skills next to it. To let several registry agents coexist, a user-scope `observal agent pull` writes each agent into its own profile directory under `~/.pi/agent/agents/{agent}/`. The Observal extension's `/agent` command swaps a profile into the live `~/.pi/agent/` location. A project-scope pull writes the rules straight to the project's `AGENTS.md`, which Pi reads directly. Pi session telemetry uses an in-process TypeScript extension, `~/.pi/agent/extensions/observal.ts`. It is installed by `observal doctor patch` and is shared by every agent; agent pulls do not embed telemetry hooks. *** ## Supported capabilities | Capability | Support | | - | - | | Agent profiles | Project and user scope, as `AGENTS.md` | | Hook bridge | Pi extension (no shell hooks) | | Extension events | `session_start`, `agent_end`, `session_shutdown` | | MCP servers | `.pi/mcp.json` and `~/.pi/agent/mcp.json` | | Agent prompt | Registry rules are written into the generated `AGENTS.md` | | Guidance files | Scanned from `AGENTS.md`, `~/.pi/agent/AGENTS.md`, `.pi/SYSTEM.md`, `.pi/APPEND_SYSTEM.md` | | Skills | `.pi/skills/{name}/SKILL.md` and `~/.pi/agent/skills/{name}/SKILL.md` | | Session parsing | Pi JSONL parser | | Telemetry | Pi session transcripts delivered through the extension; `observal reconcile --harness pi` re-pushes recent sessions found under the Pi agent directory's `sessions/` folder | | Model selection | Registry-backed Pi model catalog (`observal registry models list --harness pi`) | *** ## Setup ### 1. Install the Observal CLI ```bash theme={null} uv tool install observal-cli # or: pipx install observal-cli ``` Pi 0.74.0 or newer is required by the telemetry extension. ### 2. Authenticate ```bash theme={null} observal auth login ``` This writes credentials to `~/.observal/config.json`. Unless you pass `--no-setup` or `--output json`, login then installs the bundled Observal skills into every detected harness (Pi counts once `~/.pi/` exists) and runs `observal doctor`, which warns if the Pi telemetry extension is missing or stale. A healthy install prints no Pi warning. ### 3. Pull an agent into Pi ```bash theme={null} observal agent pull --harness pi ``` Pi's default scope is user scope. The agent is written to `~/.pi/agent/agents/{agent-name}/`. To install into the current project: ```bash theme={null} observal agent pull --harness pi --scope project ``` Project pulls write the rules to `AGENTS.md` in the project root, which Pi reads directly, and the agent's MCP servers and skills to `.pi/agents/{agent-name}/`. Pi loads project MCP servers from `.pi/mcp.json` and project skills from `.pi/skills/{name}/SKILL.md`, and `/agent` handles only user-scope profiles, so activate them by hand: ```bash theme={null} cp .pi/agents//mcp.json .pi/mcp.json cp -r .pi/agents//skills/. .pi/skills/ ``` Merge instead of copying if `.pi/mcp.json` already lists other servers. ### 4. Install or refresh the telemetry extension ```bash theme={null} observal doctor patch --harness pi ``` This writes the bundled extension to `~/.pi/agent/extensions/observal.ts` when it is missing, and refreshes a copy Observal recognises as its own when that copy has fallen behind or been edited, recording the CLI version it came from in an adjacent `.observal-extension.json`. A file Observal did not write is reported and left alone. If `npm:observal-pi` is registered in `~/.pi/agent/settings.json`, that takes precedence: nothing is installed locally, and a local copy Observal recognises as its own is removed (kept as `observal.ts.bak`) so Pi does not load the extension twice. A local file Observal did not write is left alone. Restart Pi or run `/reload` afterwards. `doctor patch` refuses to run until `observal auth login` has written a server URL, although it does not contact the server. ### 5. Activate the agent inside Pi Inside a Pi session, run `/agent` and pick the pulled agent. See [Agent profiles and swapping](#agent-profiles-and-swapping). ### 6. Check what is installed ```bash theme={null} observal scan --harness pi observal doctor ``` *** ## Config paths | Purpose | Project scope | User scope | | - | - | - | | Agent rules | `AGENTS.md` (project root) | `~/.pi/agent/agents/{agent}/AGENTS.md` | | MCP config | `.pi/agents/{agent}/mcp.json` | `~/.pi/agent/agents/{agent}/mcp.json` | | Skill definition | `.pi/agents/{agent}/skills/{name}/SKILL.md` | `~/.pi/agent/agents/{agent}/skills/{name}/SKILL.md` | | Active agent rules | `AGENTS.md` | `~/.pi/agent/AGENTS.md` | | Active MCP config | `.pi/mcp.json` | `~/.pi/agent/mcp.json` | | Active skills | `.pi/skills/{name}/SKILL.md` | `~/.pi/agent/skills/{name}/SKILL.md` | | Guidance files | `AGENTS.md`, `.pi/SYSTEM.md`, `.pi/APPEND_SYSTEM.md` | `~/.pi/agent/AGENTS.md` | | Telemetry extension | – | `~/.pi/agent/extensions/observal.ts` | | Pi settings | – | `~/.pi/agent/settings.json` | | Observal credentials | `~/.observal/config.json` | `~/.observal/config.json` | | Observal lockfile | `~/.observal/lockfile.json` | `~/.observal/lockfile.json` | | Acknowledged cursors | `~/.observal/sync_state.json` | `~/.observal/sync_state.json` | | Pending batches | `~/.observal/pi_session_outbox/` | `~/.observal/pi_session_outbox/` | Pi MCP configs use the `mcpServers` key. *** ## Agent profiles and swapping Because Pi reads a single `AGENTS.md`, `mcp.json`, and `skills/` directory, only one Observal agent can be active at a time. In user scope, `observal agent pull` does not touch the active files. It writes into the per-agent profile directory, and the extension's `/agent` command makes a profile active: 1. On first use, `/agent` backs up the current `AGENTS.md`, `SYSTEM.md`, `mcp.json`, `skills/`, and `sandboxes/` into `~/.pi/agent/agents/default/`. 2. It removes the active `AGENTS.md`, `SYSTEM.md`, `mcp.json`, `skills/`, and `sandboxes/`, then copies in whichever of those the chosen profile contains. Items the profile lacks stay absent until you swap back to `default`. 3. When Observal credentials are configured, it records the chosen agent as `active_agent` in `~/.observal/config.json` and refreshes the layer snapshot. 4. It offers to reload the session so the new system prompt takes effect. Run `/agent` with no argument to pick from installed profiles, or `/agent ` to swap directly. Choose `default` to restore the backed-up configuration and clear the active agent. `/agent` lists only user-scope profiles under `~/.pi/agent/agents/`. *** ## Extension spec Observal installs a TypeScript extension named `observal.ts`. It has no runtime dependencies beyond `node:*` built-ins and is fail-open: telemetry failures never interrupt Pi. Ingest and checkpoint calls time out after five seconds, and the layer-snapshot upload after ten. | Pi event | Observal use | | - | - | | `session_start` | Load config and cursors, upload the layer snapshot, recover stale sessions on startup, show `● observal` in the footer | | `agent_end` | Push new session lines after each turn | | `session_shutdown` | Push remaining lines and finalize the session | The extension also registers two commands, `/agent` and `/obs-sync`: | Command | Description | | - | - | | `/agent [name]` | Swap the active Observal agent profile | | `/obs-sync` | Show lines pushed and the server URL | | `/obs-sync flush` | Push pending lines now | | `/obs-sync config` | Show the config file path and server URL | `doctor patch` compares the installed file with the bundled source and replaces stale copies. *** ## Attribution Pi does not expose an Observal agent id in its session file. The extension resolves attribution from Observal's own state: 1. `observal agent pull` records the agent name, id, version, scope, pull time, and directory under the `pi` harness in `~/.observal/lockfile.json`. 2. The pull also records that agent as `active_agent` in `~/.observal/config.json`, and `/agent` updates the entry whenever you swap profiles. 3. On `session_start`, the extension reads `active_agent` and looks it up in the lockfile entry for the configured server URL, matching by id first and then by name. 4. The session payload is sent with the resolved `agent_id` and `agent_version`. 5. If there is no `active_agent`, or it has no `id`, the session is sent with a null agent id. If there is an `active_agent` but no lockfile match, the stored id is sent as-is; no directory-based guessing takes place. *** ## Session push behavior The Pi extension implements the same acknowledged delivery contract as the Python harnesses: 1. Resolve the session JSONL file and id from Pi's session manager. 2. Read complete records after the acknowledged byte and line cursor in `~/.observal/sync_state.json`. 3. Persist each pending batch under `~/.observal/pi_session_outbox/` before network delivery, in chunks of at most 500 lines. 4. Retry the batch idempotently until the server returns a contiguous acknowledgement covering it. 5. Advance the local cursor only to that acknowledged checkpoint. 6. On `session_shutdown`, send a SHA-256 audit manifest and replay any range the server asks to repair. 7. On the next startup, retry every pending outbox batch, then re-push unfinished sessions from the current project that changed within the last seven days until five have been finalized. Attempts that fail do not count toward the five. A missing or corrupt cursor is rebuilt from the server checkpoint on the next push. Pi's pending batches are the files under `~/.observal/pi_session_outbox/`. `observal ops telemetry status` reports server ingest health and the Python exporters' outbox; it does not count this directory. *** ## Agent profile format Observal writes the agent's rules as plain Markdown into `AGENTS.md`. Pi loads that file as the system prompt, so the whole file is the agent: ```markdown theme={null} # Reviewer You are a code reviewer with the following specialization... ``` There is no frontmatter and no model field. `observal agent pull` does not write a model into Pi's configuration; the model is chosen inside Pi. *** ## Skill file format Pi skills live at: | Scope | Path | | - | - | | Project | `.pi/skills/{name}/SKILL.md` | | User | `~/.pi/agent/skills/{name}/SKILL.md` | Example: ```markdown theme={null} --- description: "Runs the project test suite" --- # Run Tests Run `pytest -q` from the project root. ``` When Codex is also installed, the bundled Observal skills are written once to the shared `~/.agents/skills/{name}/SKILL.md` location, which both harnesses read, instead of being duplicated under `~/.pi/agent/skills/`. *** ## Caveats **One active agent at a time.** A user-scope pull does not activate the agent. Run `/agent` inside Pi, or copy the profile into `~/.pi/agent/` by hand. `/agent` does not see project-scope profiles under `.pi/agents/`; copy their `mcp.json` and `skills/` into `.pi/` as shown in Setup step 3. **`/agent` replaces the active files.** The first swap backs up your existing `AGENTS.md`, `SYSTEM.md`, `mcp.json`, `skills/`, and `sandboxes/` into the `default` profile. Later swaps remove those five items and install only what the chosen profile contains, so a profile without `SYSTEM.md` leaves Pi with no `SYSTEM.md`. Edit the profile directory, not the active files, if you want changes to survive a swap. **The extension is shared per Pi install.** It is installed by `doctor patch`, not by each agent pull, and lives only in user scope. **Guidance files are scanned, with two exceptions.** Observal layers `AGENTS.md`, `.pi/SYSTEM.md`, and `.pi/APPEND_SYSTEM.md` as context. Scanning and `observal agent pull` never rewrite the project's `.pi/SYSTEM.md` or `.pi/APPEND_SYSTEM.md`. A project-scope pull does write the project's `AGENTS.md`, because that file is the agent's rules in Pi, and `/agent` swaps do replace the user-scope `~/.pi/agent/SYSTEM.md` as described above. **Attribution depends on the lockfile and `/agent`.** A profile copied by hand does not update `active_agent`, so its sessions keep whatever binding `~/.observal/config.json` already holds, or carry no agent id if it has none. **MCP config is Pi-specific.** Pi uses `mcp.json` with the `mcpServers` key under `.pi/` or `~/.pi/agent/`, not Claude Code or Kiro MCP paths. # API endpoints Source: https://docs.observal.io/reference/api-endpoints REST and GraphQL surface of the Observal server. Unless noted, endpoints require authentication via Bearer token or API key (`Authorization: Bearer ` or `X-API-Key: `). Base path: `/api/v1`. ## Auth | Method | Path | Description | | - | - | - | | `POST` | `/auth/bootstrap` | Auto-create admin on fresh server (localhost only) | | `POST` | `/auth/register` | Self-registration (email + password; `deployment.sso_only=false` only) | | `POST` | `/auth/login` | Login with API key or email + password | | `POST` | `/auth/exchange` | Exchange one-time OAuth code for credentials | | `GET` | `/auth/whoami` | Current user info | | `POST` | `/auth/token` | Exchange credentials for JWT access + refresh tokens | | `POST` | `/auth/token/refresh` | Rotate refresh token for new access token | | `POST` | `/auth/token/revoke` | Revoke a refresh token | | `GET` | `/auth/oauth/login` | Initiate OAuth SSO flow | | `GET` | `/auth/oauth/callback` | OAuth callback handler | ## Registry Per type: `mcps`, `agents`, `skills`, `hooks`, `prompts`, `sandboxes`. All `{id}` parameters accept a UUID or a name. | Method | Path | Description | | - | - | - | | `POST` | `/{type}` | Submit / create | | `GET` | `/{type}` | List approved items | | `GET` | `/{type}/{id}` | Get details | | `POST` | `/{type}/{id}/install` | Get harness config snippet | | `DELETE` | `/{type}/{id}` | Delete | | `GET` | `/{type}/{id}/metrics` | Metrics | | `POST` | `/agents/{id}/pull` | Pull agent (installs all components) | `POST /{type}/{id}/install` accepts `version` for agents, MCP servers, skills, and hooks. The response reports the `version` that was installed; component installs also return its `version_id` and content `digest`. ### MCP webhook sync Owner routes manage GitHub or GitLab webhook sync for one MCP listing. See [Sync an MCP server from GitHub or GitLab](/use-cases/mcp-repo-sync). | Method | Path | Description | | - | - | - | | `GET` | `/mcps/{id}/webhook-sync` | Sync settings and last result. `enabled` is false when sync is off | | `PUT` | `/mcps/{id}/webhook-sync` | Turn on or update sync: `sync_on_push`, `sync_on_release`, `branch`, `provider` (`github` or `gitlab`; detected from the repository URL when omitted on first enable). Returns `secret` only on first enable | | `POST` | `/mcps/{id}/webhook-sync/rotate-secret` | Issue a new secret, returned once | | `POST` | `/mcps/{id}/webhook-sync/run` | Queue a sync of the tracked branch (`202`) | | `DELETE` | `/mcps/{id}/webhook-sync` | Turn off sync | | `POST` | `/webhooks/github/mcp/{sync_id}` | GitHub delivery receiver. No user token: authenticated by `X-Hub-Signature-256`. Returns `202` with `status` `queued` or `ignored` | | `POST` | `/webhooks/gitlab/mcp/{sync_id}` | GitLab delivery receiver. No user token: authenticated by `X-Gitlab-Token`. Returns `202` with `status` `queued` or `ignored` | ### Registry forks Only authenticated users can fork an approved listing and an approved base release. The result is an independently owned **draft**; no source content or counts are changed. For component endpoints, `{type}` is one of `mcps`, `skills`, `hooks`, `prompts`, or `sandboxes`. | Method | Path | Description | | - | - | - | | `POST` | `/agents/{id}/fork` or `/{type}/{id}/fork` | Create a draft fork; optional body fields: `name`, `version`, `new_version`, `team_id`, `visibility` | | `GET` | `/agents/{id}/forks` or `/{type}/{id}/forks` | Paginated, public approved direct forks only; `limit` and `offset` query parameters | | `GET` | `/agents/{id}/fork-diff?version=…` or `/{type}/{id}/fork-diff?version=…` | Unified `diff` and `unchanged` relative to the fork's approved base; defaults to the latest fork version visible to the caller | Diff access follows normal fork-version permissions. If the upstream is no longer visible or its approved base is gone, comparison returns `404`, never a saved source reference. Component diffs omit secret environment/header values and use a digest marker for install content excluded from public version responses. [Fork and customize](/use-cases/fork-and-customize) describes the workflow. ### Agent versions and locks Each agent version pins exact component versions. Installs generate every component from its pinned version. Agent versions released before pinning can have unlocked components: those install at their latest approved release, are reported with `source: fallback-latest` in the install `lock`, and are refused by a `strict` install. | Method | Path | Description | | - | - | - | | `GET` | `/agents/{id}/versions` | Version history | | `POST` | `/agents/{id}/versions` | Release a version for review. Components keep the current release's pins unless a component names a `version` or `refresh_components` is true | | `GET` | `/agents/{id}/versions/{version}/lock` | The version's lock: exact component versions, version ids, and digests. Frozen once approved | | `GET` | `/agents/{id}/versions/{version}/outdated` | For each pin, the latest approved release and whether the pin is behind | | `PUT` | `/agents/{id}` | Edits version-owned fields only while the latest version is a draft, pending, or rejected; otherwise `409` | A component reference's `version` must name an approved (or archived) release. Drafts may also pin a release that is not approved yet, but only a caller who may read it (its owner, co-authors, admins, and reviewers); to anyone else it does not exist, as on the component version routes. `POST /agents/{id}/install` also accepts `strict`. Its response includes `version` (the agent version installed) and `lock`: `status` (`locked`, `partial`, or `unlocked`), `digest`, `components` with each one's `source`, and `problems`. With `strict: true`, any problem is a `409` and nothing is generated. ### Scan | Method | Path | Description | | - | - | - | | `POST` | `/scan` | Bulk register items from harness config scan | ### Review | Method | Path | Description | | - | - | - | | `GET` | `/review` | List pending submissions | | `GET` | `/review/{id}` | Submission details | | `POST` | `/review/{id}/approve` | Approve | | `POST` | `/review/{id}/reject` | Reject | ## Discovery (ARD) Agentic Resource Discovery endpoints. Authentication is optional: anonymous callers see public, approved resources only when `deployment.public_registry_enabled` is on; authenticated callers see what the registry's visibility rules already grant them (public, their teams', and their own drafts). Errors use the ARD envelope `{"errorCode": "INVALID_ARGUMENT", "message": "..."}`. See [ADR 0001](/adr/0001-agentic-resource-discovery). | Method | Path | Description | | - | - | - | | `GET` | `/.well-known/ard.json` | Manifest of the registry entry plus this deployment's public approved resources (not imported remote agents) | | `GET` | `/.well-known/ai-catalog.json` | Predecessor path, same document | | `POST` | `/api/v1/ard/search` | ARD Search. `score` is relevance only; `obs:approval`, `obs:availability`, `obs:supportedHarnesses`, `obs:delegable` carry the rest; remote A2A results add `obs:provider` from the card | | `GET` | `/api/v1/ard/agents` | ARD List: deterministic browse with `filter`, `orderBy`, `pageSize`, `pageToken` | | `POST` | `/api/v1/ard/explore` | ARD Explore: not implemented (`501`) | | `GET` | `/api/v1/ard/entries/{identifier}` | One complete entry by `urn:air:` identifier (Observal-specific) | | `GET` | `/api/v1/artifacts/{kind}/{uuid}/{version}` | Permanent versioned artifact; `Digest` and `X-Artifact-Digest` headers | Search request body follows the spec: `{"query": {"text": "...", "filter": {...}}, "federation": "none", "pageSize": 5}`. Supported filter terms: `type`, `tags`, `capabilities`, `publisher`, `version`, `obs:kind`, `obs:supportedHarnesses`, `obs:lifecycle`, `obs:activatable`, `obs:forksOf`. The last filter takes one upstream `urn:air:` identifier and returns only currently **public approved** direct forks. Search, List, entry, and manifest serialization add `obs:forkedFrom` only while both fork and source are currently public and approved; private lineage is never projected from a saved reference. ### Remote A2A agents Remote agents are registered by Agent Card URL and reviewed before anyone can delegate to them. They then appear in search as `type: application/a2a-agent-card+json` with the reviewed card pinned in the entry (`obs:agentCard`, `obs:a2aInterface`). Clients call the agent directly; the server does not proxy A2A traffic. See [ADR 0002](/adr/0002-a2a-delegation). | Method | Path | Description | | - | - | - | | `GET` | `/api/v1/ard/imports` | Remote agents the caller can see, including their own and (reviewers) pending ones | | `POST` | `/api/v1/ard/imports/a2a` | Register or refresh a card: `{"cardUrl": "...", "visibility": "private\|team\|public", "teamId": "..."}`. `201` new, `200` refresh; a changed card returns to `pending` | | `POST` | `/api/v1/ard/imports/{identifier}/review` | Reviewers in scope (team owners and reviewers for team agents, global reviewers for public, admins for private): `{"action": "approve\|reject", "reason": "...", "digest": "sha256:..."}` (`digest` of the reviewed card required to approve, 409 when the card changed since; reason required to reject) | | `DELETE` | `/api/v1/ard/imports/{identifier}` | Owner or admin: remove from discovery | Cards must be served over https from a public address. Internal hosts are allowed individually with the `discovery.a2a_private_hosts` setting. The `discovery.delegation_enabled` setting (default on) adds the `observal-agents` MCP server to every pulled Agent. ## Telemetry | Method | Path | Description | | - | - | - | | `POST` | `/telemetry/events` | Legacy event ingestion | | `GET` | `/telemetry/status` | Data flow status | | `POST` | `/ingest/session` | Idempotently ingest indexed session source records and return the highest contiguous acknowledgement | | `GET` | `/ingest/session/checkpoint` | Get the caller's contiguous line/byte checkpoint for a harness session | Session record identity is scoped by project, user, harness, session ID, and source line index. Retrying the same content at the same index is safe; different content at an already acknowledged index returns `409`. Final session requests can include `session_hash` and `hashed_line_count`. The response includes `integrity_ok`, `server_hash`, and `repair_from_line`. A failed audit rewinds the durable checkpoint to the first affected range; the exporter then replays that range idempotently. Hashing and canonical manifest scans occur only for final/audit requests, not incremental ingest. ## Telemetry hooks | Method | Path | Description | | - | - | - | | `POST` | `/telemetry/hooks` | Ingest lifecycle hook events (used by Kiro shell hooks) | ## Alerts | Method | Path | Description | | - | - | - | | `GET` | `/alerts` | List alert rules | | `POST` | `/alerts` | Create alert rule. Non-admins may only target an agent they own or co-author | | `PATCH` | `/alerts/{id}` | Update alert rule | | `DELETE` | `/alerts/{id}` | Delete alert rule | | `GET` | `/alerts/{id}/history` | Alert firing history | ## Feedback | Method | Path | Description | | - | - | - | | `POST` | `/feedback` | Submit rating | | `GET` | `/feedback/{type}/{id}` | Get feedback | | `GET` | `/feedback/summary/{id}` | Rating summary | ## Admin Requires `admin` or `super_admin` role. | Method | Path | Description | | - | - | - | | `GET` | `/admin/settings` | List settings | | `PUT` | `/admin/settings/{key}` | Set a value | | `GET` | `/admin/users` | List users | | `POST` | `/admin/users` | Create user | | `PUT` | `/admin/users/{id}/role` | Change role | | `PUT` | `/admin/users/{id}/password` | Reset user password (admin) | | `GET` | `/admin/weights` | Get dimension weights | | `PUT` | `/admin/weights` | Set dimension weights | | `GET` | `/admin/canaries/{agent_id}/reports` | Canary detection reports | ## GraphQL Single endpoint, query + subscription via WebSocket. | Path | Description | | - | - | | `/api/v1/graphql` | Session update subscriptions | Subscriptions use `graphql-ws` protocol. ## Health | Method | Path | Description | | - | - | - | | `GET` | `/health` | Readiness - checks DB + ClickHouse | | `GET` | `/healthz` | Liveness - is the API process alive | ## Rate limiting Auth endpoints are subject to `RATE_LIMIT_AUTH` and `RATE_LIMIT_AUTH_STRICT`. Non-auth endpoints are not rate-limited by default; put a reverse proxy or API gateway in front if you need it. ## Request size limits `MAX_REQUEST_SIZE_MB` (default `10`) caps body size on all endpoints. Large telemetry batches may need tuning. ## Related * [Authentication and SSO](/self-hosting/authentication) * [Hooks specification](/reference/hooks-spec) # Config files Source: https://docs.observal.io/reference/config-files Every file Observal reads or writes on the client (`~/.observal/`) and in each harness's config directory. ## Client-side: `~/.observal/` | File | Purpose | Permissions | | - | - | - | | `config.json` | CLI config (server URL, access token, user info, timeout) | `0600` | | `aliases.json` | User-defined shortcuts (`@my-mcp` → UUID) | `0600` | | `last_results.json` | Latest typed list output, enabling row-number references | `0600` | | `telemetry_buffer.db` | Durable SQLite outbox for Python session exporters awaiting contiguous server acknowledgement | `0600` | | `opencode_session_outbox/` | Per-session durable OpenCode plugin batches and acknowledged line state | `0600` files | | `pi_session_outbox/` | Durable pending Pi extension batches | `0600` files | | `sync_state.json` | Acknowledged byte/line cursors for file-backed exporters | owner read/write | | `keys/` | Server-side JWT keys (operators only; path controlled by `JWT_KEY_DIR`) | `0600` | ### `config.json` schema ```json theme={null} { "server_url": "https://observal.your-company.internal", "access_token": "ey...", "refresh_token": "ey...", "user_id": "f9f3...", "user_name": "Alice", "username": "alice", "timeout": 30, "update_check": true, "update_check_interval": 86400, "update_check_repo": "" } ``` Authentication and identity fields are managed by `observal auth`. The `config set` command accepts only `server_url`, `timeout`, `update_check`, `update_check_interval`, and `update_check_repo`. Supported environment variables override persisted values for the current invocation; see [Environment variables](/reference/environment-variables). ### Durable session outbox Python session exporters persist each observed batch in `telemetry_buffer.db` before network delivery. OpenCode and Pi use per-session files under their native outbox directories because their TypeScript runtimes cannot call the Python SQLite engine. All follow the same protocol: pending data survives process restarts and failed attempts, and the source line advances only when the server's contiguous checkpoint covers the complete batch. Observal does not silently evict unacknowledged records at capacity. `sync_state.json` is a cache of acknowledged local positions, not the authority for delivered history. If it is missing, corrupt, or stale, recovery validates and restores positions from the authenticated server checkpoint. Finalized sessions also send a SHA-256 audit manifest; hashing is not performed on ordinary incremental uploads. Use `observal ops telemetry status` to inspect pending batch count, disk use, oldest pending time, and last successful acknowledgement. ### `aliases.json` schema ```json theme={null} { "my-mcp": "498c17ac-1234-4567-89ab-cdef01234567", "reviewer": "alice/reviewer" } ``` Use anywhere that accepts a compatible reference by prefixing the alias with `@`. ### `last_results.json` schema ```json theme={null} { "item_type": "skill", "ids": ["498c17ac-1234-4567-89ab-cdef01234567"], "names": { "reviewer": "498c17ac-1234-4567-89ab-cdef01234567" } } ``` Each list invocation replaces this cache, including an empty result. Numeric row references are valid only for the component type that produced the latest list. ## harness-side ### Claude Code | Path | Purpose | | - | - | | `~/.claude/settings.json` | Hooks, MCP servers, telemetry config | | `~/.claude/agents/.json` | User-scoped sub-agent definitions | | `.claude/agents/.json` | Project-scoped sub-agent definitions | | `.claude/skills//` | Installed skills (SKILL.md + assets) | | `AGENTS.md` / `CLAUDE.md` | Rules loaded into context | ### Kiro | Path | Purpose | | - | - | | `.kiro/settings/mcp.json` | Project-level MCP servers with direct commands or URLs | | `~/.kiro/settings/mcp.json` | Global MCP servers | | `.kiro/agents/.json` | Project-level agent config with telemetry hooks | | `~/.kiro/agents/.json` | Global agent config | | `.kiro/steering/.md` | Steering files (system instructions with YAML frontmatter for inclusion modes) | | `.kiro/skills/` | Kiro skills (SKILL.md) | | `.kiro/hooks/` | Standalone hook definitions | | `AGENTS.md` | Rules loaded into context (compat with Claude Code) | ### Cursor | Path | Purpose | | - | - | | `.cursor/mcp.json` | MCP servers with direct commands or URLs | | `.cursor/rules/` | Cursor rules | | `AGENTS.md` | Rules | ### VS Code | Path | Purpose | | - | - | | `.vscode/mcp.json` | MCP servers with direct commands or URLs | | `AGENTS.md` | Rules loaded into context | ### Codex CLI | Path | Purpose | | - | - | | `AGENTS.md` | Rules (rules-only integration) | ### Pi | Path | Purpose | | - | - | | `~/.pi/agent/extensions/observal.ts` | Bundled telemetry extension, installed locally when npm isn't configured | | `~/.pi/agent/extensions/.observal-extension.json` | Manifest recording the CLI version the local extension was installed from | | `~/.pi/agent/extensions/observal.ts.bak` | Copy of the previous extension, kept whenever Observal replaces or removes one | | `~/.pi/agent/settings.json` | Pi's own config; a `packages` entry of `npm:observal-pi[@version]` here selects the npm installation mode instead | Two mutually exclusive installation modes are supported: * **npm**: `npm:observal-pi` is configured in `~/.pi/agent/settings.json` (with or without a pinned version, and whether or not Pi has downloaded it yet). Observal never installs `extensions/observal.ts` in this mode — `observal doctor` only reports when a pinned version is older than the installed CLI, with a `pi update npm:observal-pi` reminder. Pi loads a local `extensions/observal.ts` in addition to the npm package, so if one is left over from local mode every session is sent twice; `observal doctor patch` removes a local copy it recognizes as its own (keeping it as `observal.ts.bak`) and leaves a file Observal did not write alone. * **local**: no npm package is configured. `observal auth login` and `observal doctor patch` install the extension bundled with the CLI directly to `extensions/observal.ts`, recording the installed CLI version in the adjacent manifest. A later CLI upgrade rewrites each of those files atomically — the two are written in sequence, not as a single atomic unit — and prints a reminder to restart Pi or run `/reload`; a newer local install than the current CLI is left alone. If the file still carries the current version but its contents have drifted from the bundled source — an edited working copy, say — doctor restores it and keeps the previous contents as `observal.ts.bak`. Observal keeps a copy of the file whenever it rewrites or deletes one, named `observal.ts.bak` (`.bak.1`, `.bak.2`, … if that name is taken). The copy does not end in `.ts`, so Pi does not load it; delete it once you are satisfied with the result. There are two cases where no copy is made, because nothing is lost: adopting a file whose contents already match the bundle, where only the manifest is written, and `observal doctor cleanup` removing a tracked install that still matches what Observal wrote. Every other refresh, restore, migration and duplicate removal keeps one — a manifest records which CLI version wrote an install, not that its bytes are still unedited. ### Migrating installs from before version tracking CLI versions before the manifest existed wrote `extensions/observal.ts` with nothing beside it. Those installs are recognized by the doc header every Observal extension carries, and `observal auth login` or `observal doctor patch` migrates them: the existing file is copied to `observal.ts.bak` (`.bak.1`, `.bak.2`, … if that name is taken), the current extension is written in its place, and the manifest is created. The backup does not end in `.ts`, so Pi does not load it; delete it once you're satisfied with the upgrade. If `extensions/observal.ts` exists without a matching manifest, doesn't match the bundled source, and carries no Observal header, Observal treats it as unmanaged and never overwrites or removes it — remove or move the file aside, then re-run `observal doctor patch --harness pi` to let Observal manage it. ## Safe writes `observal doctor patch` and `observal doctor cleanup` write each managed configuration file atomically and preserve unrelated entries. Use `--dry-run --output json` to review planned harness changes before writing. Agent Pull reports every created or merged file in its result. ## File permissions Client-side files under `~/.observal/` are created with mode `0600` (owner read/write only). This holds your access token, so don't loosen the permissions. ## Related * [Environment variables](/reference/environment-variables) - supported runtime overrides * [`observal config`](/cli/config), CLI surface for editing # Environment variables Source: https://docs.observal.io/reference/environment-variables Complete reference for security-relevant server and CLI environment variables. Defaults are in `.env.example` and built into the CLI. ## File-backed values Every credential listed with file support accepts `NAME_FILE=/path/to/secret` instead of `NAME=value`. Files must be UTF-8, are limited to 64 KiB, and may end with one newline. Configuring both forms is an error. The server-package installer writes generated credentials to operator-owned files under `secrets/`, with read access limited to the configured container service group. ## Server (`observal-server`) ### Core / security | Variable | Default | Description | | - | - | - | | `SECRET_KEY` | `change-me-to-a-random-string` | Application encryption secret. Supports `SECRET_KEY_FILE`. Generate a random value of at least 32 characters. | | `OLD_SECRET_KEY` | unset | Previous secret during encrypted-setting rotation. Supports `OLD_SECRET_KEY_FILE`. | | `FRONTEND_URL` | `http://localhost:3000` | External frontend URL (OAuth redirects, email links) | | `CORS_ALLOWED_ORIGINS` | `http://localhost:3000` | Comma-separated allowed CORS origins | | `MAX_REQUEST_SIZE_MB` | `10` | Maximum request body size for ordinary endpoints | | `MAX_MIGRATION_REQUEST_SIZE_MB` | `6144` | Maximum combined multipart request size for Admin migration import and validation uploads | | `RATE_LIMIT_AUTH` | `10/minute` | General auth-endpoint rate limit | | `RATE_LIMIT_AUTH_STRICT` | `5/minute` | User registration rate limit | ### Databases | Variable | Default | Description | | - | - | - | | `DATABASE_URL` | `postgresql+asyncpg://postgres:postgres@localhost:5432/observal` | PostgreSQL endpoint and source URL for `server migrate`. Supports `DATABASE_URL_FILE` in server configuration. | | `POSTGRES_USER` | `postgres` | Postgres container user | | `POSTGRES_PASSWORD` | container-specific | Direct Postgres password for existing installs | | `POSTGRES_PASSWORD_FILE` | generated by server-package setup | PostgreSQL native password-file input | | `CLICKHOUSE_URL` | `clickhouse://localhost:8123/observal` | ClickHouse endpoint and source URL for `server migrate`. Supports `CLICKHOUSE_URL_FILE` in server configuration. | | `TARGET_DATABASE_URL` | unset | Target PostgreSQL URL for `server migrate` import and validation commands | | `TARGET_CLICKHOUSE_URL` | unset | Target ClickHouse URL for `server migrate` import and validation commands | | `CLICKHOUSE_USER` | `default` | ClickHouse user | | `CLICKHOUSE_PASSWORD` | container-specific | Direct ClickHouse password for existing installs | | `REDIS_URL` | `redis://localhost:6379` | Redis connection string. Supports `REDIS_URL_FILE`. | | `DATA_RETENTION_DAYS` | `90` | ClickHouse TTL in days. `0` disables. Minimum non-zero: `7` | ### SSO OIDC, SAML, and SSO-only mode are configured in **Admin → SSO** and stored in dynamic settings. OIDC client changes require an API restart. Existing direct `OAUTH_*`, `SSO_ONLY`, and `SAML_*` values are imported once when the matching database setting is empty. File-backed forms such as `OAUTH_CLIENT_SECRET_FILE`, `GOOGLE_OAUTH_CLIENT_SECRET_FILE`, `GITHUB_OAUTH_CLIENT_SECRET_FILE`, `INSIGHTS_API_KEY_FILE`, `SAML_IDP_X509_CERT_FILE`, and `SAML_SP_KEY_ENCRYPTION_PASSWORD_FILE` remain in memory and are not imported into PostgreSQL or Redis. `SAML_SP_PRIVATE_KEY_FILE` and `SAML_SP_X509_CERT_FILE` must be configured together and override generated database SP material. ### JWT signing | Variable | Default | Description | | - | - | - | | `JWT_SIGNING_ALGORITHM` | `ES256` | `ES256` or `RS256` | | `JWT_KEY_DIR` | `~/.observal/keys` (outside Docker) / `/data/keys` (Docker) | Directory for generated signing keys. **Back this up.** | | `JWT_KEY_PASSWORD` | unset | Optional private-key encryption password. Supports `JWT_KEY_PASSWORD_FILE`. | ### AWS (Bedrock) > **Note:** Bedrock now supports [API keys](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) — generate one from the AWS console and use it like any other provider. See [Insights LLM Setup](https://github.com/Observal/Observal/blob/main/docs/insights-setup.md). These environment variables are **not required** if you use Bedrock API keys (recommended). They exist only for legacy setups using instance roles or ECS task roles where LiteLLM auto-discovers credentials from the environment. | Variable | Default | Description | | - | - | - | | `AWS_ACCESS_KEY_ID` | - | Only for legacy IAM auth (not recommended) | | `AWS_SECRET_ACCESS_KEY` | - | Only for legacy IAM auth (not recommended) | | `AWS_SESSION_TOKEN` | - | Temporary credentials (STS AssumeRole) | | `AWS_REGION_NAME` | `us-east-1` | AWS region (used by LiteLLM's boto3 client) | ### Git operations (submission analysis) | Variable | Default | Description | | - | - | - | | `ALLOW_INTERNAL_GIT_URLS` | `false` | Allow internal/private Git URLs (for GitLab / GHE) | | `GIT_CLONE_TOKEN` | - | Auth token for private repos. Supports `GIT_CLONE_TOKEN_FILE`. | | `GIT_CLONE_TOKEN_HOSTS` | - | Comma-separated exact HTTPS Git hosts allowed to receive `GIT_CLONE_TOKEN` (e.g. `github.com,gitlab.example.com`). Required for authenticated clones; add `:port` for nonstandard ports. | | `GIT_CLONE_TOKEN_USER` | `x-access-token` | Token username: `x-access-token` (GitHub), `oauth2` or `private-token` (GitLab) | | `GIT_CLONE_TIMEOUT` | `120` | Clone timeout, seconds | | `WEBHOOK_PUBLIC_URL` | - | Public base URL GitHub and GitLab use for webhook deliveries when the rest of Observal is private. Shown as the webhook URL in the MCP **Sync** tab. Unset means the deployment's normal public URL. See [Sync an MCP server from GitHub or GitLab](/use-cases/mcp-repo-sync#private-deployments) | ### Demo accounts (seeded on first startup if no users exist) | Variable | Default | | - | - | | `DEMO_SUPER_ADMIN_EMAIL` | `super@demo.example` | | `DEMO_SUPER_ADMIN_PASSWORD` | `super-changeme` in source Compose; generated file in server package | | `DEMO_ADMIN_EMAIL` | `admin@demo.example` | | `DEMO_ADMIN_PASSWORD` | `admin-changeme` in source Compose; generated file in server package | | `DEMO_REVIEWER_EMAIL` | `reviewer@demo.example` | | `DEMO_REVIEWER_PASSWORD` | `reviewer-changeme` in source Compose; generated file in server package | | `DEMO_USER_EMAIL` | `user@demo.example` | | `DEMO_USER_PASSWORD` | `user-changeme` in source Compose; generated file in server package | Each demo password supports its corresponding `_FILE` form. Remove demo variables and files before a real deployment. ### Docker host ports Used only by Docker Compose. Prometheus and Grafana ports apply only when `docker-compose.observability.yml` is included. Remap if a default is already in use. | Variable | Default | Service | | - | - | - | | `OBSERVAL_BIND_ADDRESS` | `127.0.0.1` in the server package | Host address for published ports. A non-loopback value is explicit remote exposure. | | `OBSERVAL_SECRET_GID` | operator group from `setup.sh` | Supplementary container group allowed to read server-package files with mode `0640`. | | `API_HOST_PORT` | `8000` | API (internal, behind LB) | | `WEB_HOST_PORT` | `3000` | Web UI | | `POSTGRES_HOST_PORT` | `5432` | Postgres | | `CLICKHOUSE_HOST_PORT` | `8123` | ClickHouse | | `REDIS_HOST_PORT` | `6379` | Redis | | `PROMETHEUS_HOST_PORT` | `9090` | Prometheus, optional | | `GRAFANA_HOST_PORT` | `3001` | Grafana, optional | ### Grafana Only used when the optional Grafana overlay or Terraform `observability_stack = "grafana"` is enabled. | Variable | Default | Description | | - | - | - | | `GRAFANA_ADMIN_USER` | `admin` | Grafana admin username | | `GRAFANA_ADMIN_PASSWORD` | `admin` | Direct Grafana admin password for existing installs | | `GRAFANA_ADMIN_PASSWORD_FILE` | generated file in server package | Grafana administrator password file | | `GRAFANA_CLICKHOUSE_PASSWORD_FILE` | generated file in server package | ClickHouse datasource password file mounted only into Grafana | ## CLI (`observal-cli`) Read from the environment at invocation time. Override values in `~/.observal/config.json` per invocation. | Variable | Default | Description | | - | - | - | | `OBSERVAL_SERVER_URL` | from `~/.observal/config.json` | Server URL | | `OBSERVAL_ACCESS_TOKEN` | from `~/.observal/config.json` | Access token (preferred for CI) | | `OBSERVAL_API_KEY` | from `~/.observal/config.json` | API key alias for `OBSERVAL_ACCESS_TOKEN` | | `OBSERVAL_TOKEN` | from `~/.observal/config.json` | CI token alias with highest legacy precedence | | `OBSERVAL_PASSWORD` | unset | Password for credential or bootstrap login | | `OBSERVAL_CURRENT_PASSWORD` | unset | Current password for non-interactive password changes | | `OBSERVAL_NEW_PASSWORD` | unset | New password for non-interactive or mandatory password changes | | `OBSERVAL_TIMEOUT` | `30` | HTTP timeout in seconds | The token and password variables support corresponding `_FILE` forms. Configuring both a direct variable and its file form is an error. When multiple token aliases are present, the legacy precedence remains `OBSERVAL_TOKEN`, then `OBSERVAL_API_KEY`, then `OBSERVAL_ACCESS_TOKEN`. Example CI usage: ```bash theme={null} export OBSERVAL_SERVER_URL=https://observal.your-company.internal export OBSERVAL_API_KEY='replace-with-your-key' observal ops traces --limit 100 --output json | jq ``` ## Related * [Self-Hosting → Configuration](/self-hosting/configuration), narrative view, grouped by concern * [Config files](/reference/config-files), `~/.observal/` file layout # Hooks specification Source: https://docs.observal.io/reference/hooks-spec The schema Observal uses for hook definitions -- both the registry hook type (`observal registry hook`) and hooks wired into harness configs by `observal agent pull` / `observal doctor patch`. Claude Code session hooks currently use `HOOKS_SPEC_VERSION = "12"` in `observal_cli/harness_specs/claude_code_hooks_spec.py`. Other harnesses have their own hook specs. ## Where hooks live Two distinct things share the name "hook": 1. **Registry hooks**: packaged, versioned hook definitions in the Observal registry. Install them via `observal registry hook install`. 2. **harness hooks**: entries in `~/.claude/settings.json`, `.kiro/agents/.json`, etc. These are written by `observal agent pull` and `observal doctor patch`. Both use the same event vocabulary. ## Events | Event | When it fires | | - | - | | `SessionStart` | New harness session begins | | `Stop` | Session ends | | `SubagentStop` | Sub-agent session ends (Claude Code only) | | `UserPromptSubmit` | User submits a prompt | | `PreToolUse` | Before a tool call | | `PostToolUse` | After a tool call (with result) | | `Notification` | harness surfaces a notification | Source: `observal_cli/constants.py:VALID_HOOK_EVENTS`. ## Handler types | Type | Payload | Used by | | - | - | - | | `command` | Shell command with templated args | Kiro (shell hooks only), Claude Code (local scripts) | | `http` | URL + method + headers + body | Claude Code (native HTTP hooks) | ## Execution modes | Mode | Semantics | | - | - | | `async` | Fire and forget - harness doesn't wait | | `sync` | harness waits for handler to return before continuing | | `blocking` | Handler can veto the event (e.g. block a tool call) | Source: `observal_cli/constants.py:VALID_HOOK_EXECUTION_MODES`. ## Scopes | Scope | Effect | | - | - | | `agent` | Applies only to one agent | | `session` | Applies for the duration of a session | | `global` | Applies across everything | ## Session delivery hooks Telemetry is delivered by the session push hook, which reads the local harness transcript and sends session records to `/api/v1/ingest/session`. It uses the CLI configuration in `~/.observal/config.json`; it does not require a `curl` wrapper, a telemetry environment variable, or an API key header. For Claude Code, `observal doctor patch --harness claude-code` writes **two** Observal-managed command hooks into `~/.claude/settings.json`: `UserPromptSubmit` and `Stop`. Each runs the installed Python interpreter with `-m observal_cli.hooks.session_push --harness claude-code`. Later patches recognize Observal's own hooks by their command, so user-authored hooks are left alone. For Kiro, the current IDE/CLI reads `.kiro/hooks/observal.json` with `UserPromptSubmit` and `Stop` triggers. Legacy CLI 2.x (without the IDE) uses inline `userPromptSubmit` and `stop` hooks on its agent profile. Both use `-m observal_cli.hooks.session_push --harness kiro`, not `curl`. See `observal_cli/harness_specs/kiro_hooks_spec.py` for the exact formats. Other harnesses have their own capability-specific specs. Use `observal doctor` to inspect the current installation and `observal doctor patch --all-harnesses` to instrument detected harnesses. ## Event name mapping (Claude Code ↔ Kiro) Legacy Kiro CLI 2.x inline hooks use camelCase / lowercase event names; current standalone Kiro hooks use PascalCase triggers, like Claude Code. Observal maps the legacy names as follows: | Claude Code | Kiro | | - | - | | `SessionStart` | `agentSpawn` | | `Stop` | `stop` | | `SubagentStop` | *(no equivalent)* | | `UserPromptSubmit` | `userPromptSubmit` | | `PreToolUse` | `preToolUse` | | `PostToolUse` | `postToolUse` | | `Notification` | *(no equivalent)* | ## Registry hook payload shape When submitting a hook to the registry (`observal registry hook submit`): ```json theme={null} { "name": "pretooluse-logger", "description": "Logs every tool call to a local file", "event": "PreToolUse", "handler_type": "command", "command": "echo \"$TOOL_NAME $(date)\" >> ~/.observal/tool-log.txt", "execution_mode": "async", "scope": "agent", "harness": ["claude-code", "kiro"] } ``` Each field is validated server-side against the lists in `observal_cli/constants.py` (mirrored from `observal-server/schemas/constants.py`). ## Source of truth * `observal_cli/harness_specs/`: per-harness session hook specifications and versions * `observal_cli/constants.py`: valid events, handler types, execution modes, scopes * `observal-server/schemas/constants.py`: server-side mirror A sync test (`tests/test_constants_sync.py`) ensures CLI and server stay in lockstep. ## Related * [`observal registry hook`](/cli/registry) * [Session tracking and reconciliation](/core-concepts/session-tracking) # Security assurance case Source: https://docs.observal.io/security/assurance-case ## Claim Observal protects registry control, authentication credentials, agent configuration, and telemetry against unauthorized disclosure or modification when operators follow the documented deployment requirements. This case records the threats, boundaries, controls, evidence, and residual risks supporting that claim. ## Assets The primary assets are: * User passwords, access tokens, refresh tokens, OAuth secrets, Git tokens, and signing keys * Registry agents and their MCP, skill, hook, prompt, and sandbox configurations * Session transcripts, traces, insights, audit events, and user identity data * Release artifacts, release tags, and the software supply chain * Administrative control over users, teams, policy, retention, and SSO ## Threat actors and assumptions Relevant actors include unauthenticated internet clients, malicious authenticated users, compromised user tokens, hostile registry submissions, compromised dependencies, and network attackers between clients and a remote deployment. Administrators and host operators are trusted to protect the host, secret files, backups, and TLS termination. PostgreSQL, ClickHouse, Redis, the API, and workers are trusted only inside the deployment network. A host-level compromise, malicious administrator, compromised identity provider, or compromised GitHub release workflow is outside the controls of the application alone. Recovery procedures and external infrastructure controls must address those events. ## Trust boundaries 1. **Client to frontend:** Untrusted browsers, CLIs, harness hooks, and integrations cross the HTTPS boundary into nginx and the API. Packaged HTTP listeners bind to loopback by default. Remote access requires an operator-selected TLS proxy or an explicit non-loopback bind. 2. **API authentication and authorization:** Requests cross from unauthenticated parsing into JWT, password, SSO, role, team, and ownership checks. 3. **Registry execution inputs:** URLs, archives, MCP metadata, prompts, and harness configuration cross from users or external repositories into scanners and config generation. 4. **Application to data services:** The API and workers cross the container network into PostgreSQL, ClickHouse, and Redis. Database ports bind to loopback in the server package by default. 5. **Secret storage:** Operator-controlled secret files and JWT or SAML key files cross into application memory. File-backed dynamic secrets are not copied into PostgreSQL or Redis. 6. **External services:** OAuth providers, source repositories, webhooks, and model providers cross outbound HTTPS and SSRF policy boundaries. 7. **Release supply chain:** Source and GitHub Actions identities cross into signed tags, built artifacts, provenance attestations, and public downloads. ## Security requirements and arguments ### Identity and least privilege Passwords use scrypt with a per-user salt. Tokens use configurable ES256 or RS256 asymmetric signing, short-lived access tokens, refresh-token revocation, and published JWKS. Role, team, visibility, ownership, and admin dependencies enforce authorization at API boundaries. Redis-backed revocation fails closed. Evidence: [`models/user.py`](https://github.com/Observal/Observal/blob/main/observal-server/models/user.py), [`services/crypto.py`](https://github.com/Observal/Observal/blob/main/observal-server/services/crypto.py), and [Authentication and SSO](/self-hosting/authentication). ### Credential and key protection Operator credentials support bounded `NAME_FILE` inputs, direct and file values cannot be configured together, and server-package secrets are operator-owned and readable only by the configured container service group. JWT keys are persisted separately with owner-only permissions. SAML SP keys and certificates can be mounted as dedicated files. File-backed admin values are reported as externally managed and cannot be overwritten or revoked through the admin API. End-user password hashes and issued-token metadata remain in their purpose-built database records. They are not deployer secrets and do not belong in shared host secret files. Evidence: [`observal_shared/secrets.py`](https://github.com/Observal/Observal/blob/main/packages/observal-shared/observal_shared/secrets.py), [Configuration](/self-hosting/configuration), and [Environment variables](/reference/environment-variables). ### Network confidentiality Remote deployments are expected to terminate TLS 1.2 or TLS 1.3 using Caddy, nginx, an enterprise load balancer, or a cloud ingress. Standard HTTPS clients retain certificate and hostname verification. The server package exposes HTTP and data-service ports only on loopback by default. A non-loopback plaintext listener requires an explicit bind setting. Evidence: [Single-node deployment](/self-hosting/single-node-deploy), [`docker/nginx.production.conf`](https://github.com/Observal/Observal/blob/main/docker/nginx.production.conf), and [Requirements](/self-hosting/requirements). ### Untrusted input containment Pydantic schemas, enums, length bounds, allowlists, namespace validation, and database constraints reject malformed input. Outbound URL paths use SSRF guards. Redirects, archive paths, XML, SVG, host headers, proxy headers, and request sizes receive dedicated validation. Containers add non-root execution, read-only filesystems, temporary filesystems, and `no-new-privileges` where supported. Evidence: [Trusted proxies and network security](/self-hosting/trusted-proxies), [`services/ssrf_guard.py`](https://github.com/Observal/Observal/blob/main/observal-server/services/ssrf_guard.py), and [`docker/docker-compose.yml`](https://github.com/Observal/Observal/blob/main/docker/docker-compose.yml). ### Cryptographic and supply-chain integrity JWT signing can switch between ES256 and RS256 without recompilation. Algorithm changes retain old public verification keys for the token transition window and reject header or key-type confusion. Release artifacts carry GitHub keyless Sigstore provenance, and release tags are signed and verified with the release workflow's OIDC identity before publication. Evidence: [`services/crypto.py`](https://github.com/Observal/Observal/blob/main/observal-server/services/crypto.py), [Release verification](/security/release-verification), and [the release workflow](https://github.com/Observal/Observal/blob/main/.github/workflows/release.yml). ### Detection and recovery Security-sensitive administration emits audit and security events. Health checks, diagnostics, backup guidance, key backup guidance, retention controls, and vulnerability reporting support detection and recovery. Private vulnerability reports have documented acknowledgement and remediation targets. Evidence: [Security policy](https://github.com/Observal/Observal/blob/main/SECURITY.md), [Backup and restore](/self-hosting/backup-and-restore), and [Authentication and SSO](/self-hosting/authentication). ## Common implementation weakness mitigations | Weakness | Mitigation | | - | - | | Injection and malformed data | Typed schemas, allowlists, parameterized SQLAlchemy queries, and output encoding | | Broken access control | Central role dependencies plus ownership, team, and visibility checks | | SSRF | Scheme and address validation before outbound requests | | Path traversal and unsafe archives | Resolved-path containment, archive member limits, and type checks | | Credential disclosure | Secret redaction, encrypted sensitive settings, restricted secret files, and support-bundle filtering | | Token forgery or algorithm confusion | Asymmetric signatures, key-derived allowlists, required key IDs, expiry, and revocation | | XML and browser attacks | SAML validation, XML protections, SVG sanitization, CSP, frame denial, and content-type controls | | Resource exhaustion | Request, line, archive, query, timeout, connection-pool, and rate limits | | Supply-chain substitution | Locked dependencies, dependency scanning, signed release tags, checksums, and artifact provenance | ## Residual risks * Explicit non-loopback plaintext mode exposes credentials and telemetry to network interception. It is intended only for trusted private networks or an upstream TLS terminator. * Loopback HTTP relies on the host boundary. Other privileged processes on a compromised host can observe local traffic and files. * File-backed secrets are only as secure as host permissions, mounts, backups, and the operator's rotation process. * A compromised administrator, host, identity provider, or canonical GitHub Actions workflow can exercise its trusted authority. * Third-party MCP servers, hooks, skills, models, and harnesses may process data outside Observal's boundary. Operators must review and isolate them appropriately. * Historical releases and tags created before signed-tag enforcement may not have equivalent evidence. ## Maintenance Review this assurance case when authentication, authorization, cryptography, deployment exposure, secret storage, release signing, or a trust boundary changes. Security reports may invalidate an argument and must be reflected here after remediation. # Verify an Observal release Source: https://docs.observal.io/security/release-verification Observal release artifacts have GitHub keyless Sigstore provenance attestations. Release tags created after this policy was introduced are signed separately with gitsign. These checks prove different things: * Artifact verification binds a downloaded file and its SHA-256 digest to the Observal release workflow. * Tag verification binds a Git tag to the same GitHub Actions workflow identity. Historical tags may be unsigned. Never treat an artifact checksum alone as proof of origin. ## Download the release For a release produced by the release-branch pipeline, replace `v1.14.0` and `release/1.14` below with the exact version and minor line you want to verify: ```bash theme={null} gh release download v1.14.0 --repo Observal/Observal --dir observal-release cd observal-release ``` ## Check the downloaded bytes The release includes `checksums.txt`: ```bash theme={null} sha256sum --check checksums.txt ``` On macOS, use `shasum -a 256 --check checksums.txt`. ## Verify artifact provenance Install the [GitHub CLI](https://cli.github.com/) and verify each artifact you intend to run: ```bash theme={null} gh attestation verify ./observal-linux-x64 \ --repo Observal/Observal \ --bundle ./build-provenance.intoto.jsonl \ --signer-workflow Observal/Observal/.github/workflows/release.yml@refs/heads/release/1.14 ``` Use the downloaded server archive or another CLI binary in place of `observal-linux-x64`. A successful result verifies the artifact digest, Sigstore certificate chain, source repository, and release workflow identity. GitHub keyless signing uses short-lived credentials issued to the workflow. Observal has no long-lived release private key stored on GitHub Releases or another download site. ## Verify a release tag Install [gitsign](https://github.com/sigstore/gitsign), clone the repository, and fetch the tag: ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal git fetch origin tag v1.14.0 gitsign verify-tag v1.14.0 \ --certificate-identity https://github.com/Observal/Observal/.github/workflows/release.yml@refs/heads/release/1.14 \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` The expected Fulcio certificate identity is the release workflow on the exact matching `refs/heads/release/X.Y` branch. The expected OIDC issuer is GitHub Actions. Verification also checks the signature's transparency-log evidence. A signed tag does not replace artifact provenance verification. Verify both when establishing the source commit and the exact downloaded bytes matters. Historical releases made before the release-branch migration used `refs/heads/main` as their signing identity. Verify those historical artifacts against that original identity explicitly. The new publishing workflow never falls back to the historical identity or accepts an unsigned tag. # Authentication and SSO Source: https://docs.observal.io/self-hosting/authentication How Observal authenticates users and signs tokens, and how to wire up SSO. ## Authentication modes Observal supports password auth, API keys, OAuth / OIDC, and SAML. The public login page shows only the methods enabled for the deployment. | Method | How it is enabled | Notes | | - | - | - | | Email + password | Default password auth | Used by bootstrap admins and locally managed users | | Self registration | `auth.self_registration_enabled=true` | Creates standard `user` accounts only | | OAuth / OIDC | SSO tab settings: `oauth.client_id`, `oauth.client_secret`, `oauth.server_metadata_url` | Uses IdP discovery metadata. API restart required after changes. | | SAML | SSO tab settings: `saml.*` | SAML setup with JIT provisioning | | API keys | User generated after login | Inherits the user's role | Use `deployment.sso_only=true` when password login should be hidden and only SSO should be available. ## SSO-only mode `deployment.sso_only` controls whether password-based authentication is available. | Value | Effect | | - | - | | `false` (default) | Email and password login stays available alongside SSO. | | `true` | Password login, password reset, and local user bootstrap are blocked. Users must sign in with OIDC or SAML. | Set it in **Admin → SSO → Access policy**. Confirm OIDC or SAML works with a real test user before enabling it. ## Self registration Controls whether visitors can create their own Observal account from the login page. | Value | Effect | | - | - | | `true` | Shows a **Register** button on the login page and allows public account creation | | `false` (default) | Hides registration and blocks `POST /api/v1/auth/register` | You can set this in the web UI at **Admin → Settings → Authentication → Self Registration Enabled**. If you prefer the CLI, set the same dynamic setting directly: ```bash theme={null} observal admin set auth.self_registration_enabled true ``` New accounts are created with the built-in `user` role. They cannot review submissions, manage users, or change server settings unless an admin promotes them later. Disable it again with: ```bash theme={null} observal admin set auth.self_registration_enabled false ``` ## The bootstrap flow On a fresh server with no users, the `/api/v1/auth/bootstrap` endpoint is available **to localhost only**. When you run `observal auth login`, the CLI detects the empty user table and bootstraps an admin account interactively. This is how you create the first admin without any pre-existing credential. Once the first admin exists, bootstrap is disabled. ## JWT signing keys Tokens are signed with asymmetric keys (ES256 by default, RS256 also supported). Keys are generated on first startup and stored in the `apidata` volume at `$JWT_KEY_DIR` (default `/data/keys`). ``` JWT_SIGNING_ALGORITHM=ES256 # ES256 (default) or RS256 JWT_KEY_DIR=/data/keys ``` ### Critical: back up `$JWT_KEY_DIR` Losing these keys invalidates **every** access and refresh token. All users must log in again. Tokens rotate, but only the private key can issue new ones, and there is no recovery path without the keys. Back up the `apidata` volume every time you back up Postgres. See [Backup and restore](/self-hosting/backup-and-restore). ### Key rotation and algorithm changes For a same-algorithm emergency rotation, stop the API, back up and remove `signing.pem`, then restart. New keys are generated and existing sessions must authenticate again. To move between ES256 and RS256, change `JWT_SIGNING_ALGORITHM` and restart. Observal archives the old public key, generates the selected key type, publishes both in JWKS, and continues verifying old tokens until they expire. New tokens use only the configured algorithm. Unsupported algorithms and token headers that do not match the resolved key type are rejected. If the signing key is encrypted, use `JWT_KEY_PASSWORD_FILE` and rotate the password and key together during a planned restart. ### File-backed SAML keys SAML service-provider keys can remain outside the settings database: ```dotenv theme={null} SAML_SP_PRIVATE_KEY_FILE=/run/secrets/saml_sp_private_key SAML_SP_X509_CERT_FILE=/run/secrets/saml_sp_x509_cert ``` Both files are required together. File-backed material overrides the generated database key, appears as externally managed in admin responses, and cannot be replaced through the admin API. Replace both files atomically and restart the API to rotate them. OAuth, Google, GitHub, and SAML secret environment imports also accept the `NAME_FILE` form. File-backed values stay in memory rather than being copied into PostgreSQL or Redis. See [Configuration](/self-hosting/configuration#secret-files). ## OAuth / OIDC SSO Set these three in **Admin → SSO → SSO settings**, then restart the API so the OIDC client is rebuilt: | Setting | Value | | - | - | | `oauth.client_id` | Client ID from your IdP | | `oauth.client_secret` | Client secret from your IdP | | `oauth.server_metadata_url` | OIDC discovery URL, for example `https://accounts.example.com/.well-known/openid-configuration` | Observal uses [Authlib](https://docs.authlib.org/) and reads the IdP discovery document, so any OIDC-compliant provider works (Auth0, Okta, Azure AD, Google Workspace, Keycloak, Authentik, Dex, etc.). ### OIDC Client ID The public client identifier from your IdP application registration. ### OIDC Client Secret The private client secret from your IdP application registration. It is stored encrypted and is never shown again after saving. ### OIDC Discovery URL The `.well-known/openid-configuration` URL for your IdP tenant or authorization server. ### Redirect URI Configure your IdP to allow: ``` {FRONTEND_URL}/api/v1/auth/oauth/callback ``` With `FRONTEND_URL=https://observal.your-company.internal`, that's: ``` https://observal.your-company.internal/api/v1/auth/oauth/callback ``` ### First OAuth login The first user who logs in via OAuth is **not** automatically an admin. Bootstrap a local admin first (via `observal auth login` before enabling OAuth, or via the demo super admin), then use that admin to promote the OAuth user. ### Scope / claims Observal requests standard `openid profile email` scope. The IdP's `email` claim is the canonical user identifier. ## Google OAuth (first-class provider) Google sign-in runs as its own provider, separate from the generic OIDC slot above. Both can be enabled at the same time, so a deployment can offer Okta *and* Google on the login screen. Set these in the SSO settings page, or set them as container env vars for one-time import at startup. The **Sign in with Google** button appears after the API restarts: ``` GOOGLE_OAUTH_CLIENT_ID=1234567890-abc...apps.googleusercontent.com GOOGLE_OAUTH_CLIENT_SECRET=GOCSPX-... ``` The Google OIDC discovery URL is hardcoded server-side, so you don't need to set it. ### Creating the Google OAuth client 1. Open the [Google Cloud Console](https://console.cloud.google.com/apis/credentials) in the project you want to use. 2. Click **Create Credentials → OAuth client ID**. 3. **Application type:** Web application. 4. **Authorized JavaScript origins:** `{FRONTEND_URL}` (e.g. `https://observal.your-company.internal`). 5. **Authorized redirect URI:** `{FRONTEND_URL}/api/v1/auth/oauth/google/callback`. 6. Copy the generated **Client ID** and **Client secret** into SSO settings or your container env. 7. Restart the API container so the Authlib client is rebuilt. ### Restricting to specific email domains Set `GOOGLE_OAUTH_ALLOWED_DOMAINS` to a comma-separated list of domains. Anyone outside the list is rejected with a 403, even if they have a valid Google account. ``` GOOGLE_OAUTH_ALLOWED_DOMAINS=acme.com,acme.io ``` Leave it unset to allow any Google account (including personal `@gmail.com` addresses) to provision themselves as `role=user`. ### Notes * Observal additionally requires Google's `email_verified` claim to be `true`. Unverified accounts (rare on Google but possible) are rejected with a 400. * The first Google user is **not** automatically an admin (matches the generic OIDC behavior). Bootstrap a local admin first, then use that account to promote the Google user. * The auth provider and Google subject ID are recorded on the user row (`auth_provider="google"`, `sso_subject_id=`) for audit purposes. ## GitHub OAuth (first-class provider) GitHub sign-in runs as its own provider, alongside generic OIDC and Google. Unlike those two, GitHub is plain OAuth 2.0 (not OIDC): there is no discovery URL and no ID token. Observal fetches the profile and email list from the GitHub REST API after the code exchange. Set these in the SSO settings page, or as container env vars for one-time import at startup. The **Sign in with GitHub** button appears after the API restarts: ``` GITHUB_OAUTH_CLIENT_ID=Iv1.abc123... GITHUB_OAUTH_CLIENT_SECRET=ghp_... / github_pat_... style secret ``` ### Creating the GitHub OAuth app 1. Open **Settings → Developer settings → OAuth Apps** (on your personal account or, preferably, your GitHub organization) and click **New OAuth App**. 2. **Homepage URL:** `{FRONTEND_URL}` (e.g. `https://observal.your-company.internal`). 3. **Authorization callback URL:** `{FRONTEND_URL}/api/v1/auth/oauth/github/callback`. 4. Copy the generated **Client ID** and generate a **Client secret**; put both into SSO settings or your container env. 5. Restart the API container so the Authlib client is rebuilt. ### Restricting to GitHub organizations Set `GITHUB_OAUTH_ALLOWED_ORGS` to a comma-separated list of GitHub organization slugs. Only *active* members of at least one listed org can sign in; everyone else is rejected with a 403. Pending invitations don't count. ``` GITHUB_OAUTH_ALLOWED_ORGS=acme-inc,acme-labs ``` When an org allowlist is configured, Observal requests the `read:org` scope (in addition to `read:user user:email`) so it can see private org memberships. Members may need to grant/request org approval for the OAuth app if the org restricts third-party access. Leave it unset to allow any GitHub account to provision itself as `role=user`. ### Notes * Observal only accepts **verified** email addresses from `GET /user/emails` (preferring the primary). The profile-level `email` field is never trusted, and accounts with no verified email are rejected with a 400. * Departments are not populated automatically from GitHub (there is no groups claim). Assign them in **Admin → Users**, individually or via bulk upload — same as Google users. * The auth provider and the *numeric* GitHub user ID are recorded on the user row (`auth_provider="github"`, `sso_subject_id=`). The numeric ID is used instead of the login handle because handles can be renamed and re-registered. ## Role-based access control (RBAC) Four built-in roles enforced on every endpoint: | Role | Typical abilities | | - | - | | `user` | Publish components, install agents, view their own data | | `reviewer` | + approve/reject registry submissions | | `admin` | + manage users, change server settings | | `super_admin` | + sensitive super-admin-only operations | Change a user's role: ```bash theme={null} observal admin users # GET /api/v1/admin/users/{id}/role to inspect # PUT /api/v1/admin/users/{id}/role to change ``` Or in the web UI at `/settings/users`. ## API keys Users can generate API keys for scripts and CI. The key inherits the user's role. ```bash theme={null} # Get a key - flow depends on your setup (web UI usually) # Then in CI: export OBSERVAL_API_KEY='replace-with-your-key' export OBSERVAL_SERVER_URL=https://observal.your-company.internal observal ops traces --limit 100 --output json | jq ``` Keys can be revoked via `POST /api/v1/auth/token/revoke`. ## Rate limits Auth endpoints are rate-limited to slow brute-force attempts: | Setting | Default | Scope | | - | - | - | | `RATE_LIMIT_AUTH` | `10/minute` | General auth endpoints | | `RATE_LIMIT_AUTH_STRICT` | `5/minute` | User registration | Tighten for public-facing deployments. ## Password reset There is no self-service reset flow. The login page says **Forgot password? Contact your admin.** — an admin resets the account from **Admin > Users** in the web UI or with the CLI: ```bash theme={null} observal admin reset-password observal admin reset-password --generate ``` `--generate` has the server mint a random password and the CLI prints it once. Hand it to the user out-of-band (Slack, phone); they must change it at their next sign-in. ## Operational controls Observal includes: * **Audit logging**: every privileged action lands in ClickHouse's `audit_log` * **SSO-only mode** (`deployment.sso_only=true`) See `docs/self-hosting/sso-cli.md` for SSO CLI commands. ## Next → [Session tracking and reconciliation](/core-concepts/session-tracking) # AWS deployment with Terraform Source: https://docs.observal.io/self-hosting/aws-terraform End state: an Observal install running in your own AWS account, fronted by an Application Load Balancer with HTTPS, with managed Postgres + Redis, ECS Fargate for the stateless app tier, and a single EC2 host for ClickHouse. Prometheus and Grafana are optional through `observability_stack`. This is the recommended path for enterprise self-hosting on AWS. If you only want to evaluate Observal, use [Docker Compose setup](/self-hosting/docker-compose) instead. ## What gets provisioned A single `terraform apply` creates: * **VPC** with public + private subnets across two availability zones, NAT gateway, VPC flow logs * **Application Load Balancer** with HTTPS (ACM certificate, DNS-validated) when you supply a domain; HTTP-only otherwise. Path-based rules: `/api/*` → api service, optional `/grafana/*` → Grafana, default → web * **ECS Fargate cluster** running: * `api` (FastAPI): 2 tasks by default, autoscales 2–10 on CPU * `web` (Vite): 2 tasks by default, autoscales 2–6 on CPU * `worker` (arq background jobs): 1 task by default, autoscales 1–5 on CPU * `init` (one-shot migrations + seeds): runs as a Fargate `RunTask` whenever `image_tag` changes * **RDS Postgres 16**: Multi-AZ on `prod`, encrypted, automated daily backups, Performance Insights, Enhanced Monitoring, log exports * **ElastiCache Redis 7**: 2-node replication group with automatic failover on `prod`, slow-log to CloudWatch * **Data tier EC2** (Amazon Linux 2023): single host running ClickHouse on EBS gp3, optional Prometheus and Grafana, ENI with static private IP, internal Route 53 zone for DNS, daily ClickHouse → S3 snapshot via systemd timer * **S3 backups bucket**: versioned, AES256, lifecycle to STANDARD\_IA → GLACIER\_IR → expire, TLS-only * **CloudWatch log groups**: per ECS service, data host, RDS, Redis slow log, VPC flow logs * **SSM Parameter Store**: generated DB / ClickHouse / SECRET\_KEY / optional Grafana passwords, plus pre-built connection URLs injected into ECS tasks * **SSM Session Manager**: shell access to the data host, no SSH ClickHouse runs on EC2. The data volume keeps it durable across instance replacements; ClickHouse HA is out of scope for this module. ## Prerequisites | Requirement | Why | | - | - | | AWS account with billing enabled | obvious | | Terraform ≥ 1.6 | `brew install terraform` or use [tenv](https://tofuutils.github.io/tenv/) | | AWS CLI v2 | `brew install awscli` - also used by the one-shot init task runner | | [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) | shell access into the data host | | IAM principal with sufficient rights | see [Required IAM permissions](/self-hosting/aws-terraform#required-iam-permissions) | | (Optional) Route 53 hosted zone | required for HTTPS on a custom domain | | (Recommended) S3 bucket + DynamoDB table | remote Terraform state - see [Remote state](/self-hosting/aws-terraform#remote-state) | ## Quickstart ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal/infra/terraform/aws # 1. Authenticate to AWS (any of these works) export AWS_PROFILE=observal-prod # or: aws configure sso # or: export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... # 2. Configure cp terraform.tfvars.example terraform.tfvars $EDITOR terraform.tfvars # 3. Apply terraform init terraform plan -out tf.plan terraform apply tf.plan ``` First-time apply takes 12–15 minutes. RDS provisioning dominates. When it finishes: ```bash theme={null} terraform output app_url ``` Open that URL in your browser. The API and web tasks start a couple of minutes after the ALB targets register; refresh until you see the Observal login page. A ready-to-apply call of the module lives at [`infra/terraform/aws/examples/minimal`](https://github.com/Observal/Observal/blob/main/infra/terraform/aws/examples/minimal/README.md). ## Configuration All inputs live in `terraform.tfvars`. The defaults are production-shaped; you can apply with very little changed. ### Minimal configuration (HTTP, no custom domain) ```hcl theme={null} region = "us-east-1" environment = "prod" ``` The install comes up on the ALB's AWS-assigned hostname (e.g. `observal-prod-alb-1234.us-east-1.elb.amazonaws.com`) over plain HTTP. Useful for evaluation; not for production. ### Recommended configuration (HTTPS on your domain) ```hcl theme={null} region = "us-east-1" environment = "prod" sizing = "medium" domain_name = "observal.example.com" route53_zone_id = "Z0123456789ABCDEFGHIJ" # Lock down ingress to your corporate egress / VPN alb_ingress_cidrs = ["203.0.113.0/24", "198.51.100.0/24"] ``` Terraform requests an ACM certificate, validates it via DNS records in your hosted zone, attaches it to the ALB, and creates the alias A record pointing the domain at the ALB. There is no manual DNS step. ### Sizing presets Instead of tuning 12+ resource variables, pick a preset: ```hcl theme={null} sizing = "medium" # small | medium | large | custom ``` | Preset | \~\$/mo | API tasks | Web tasks | Worker tasks | Data host | RDS | Redis | | - | - | - | - | - | - | - | - | | `small` | \$150 | 1× (0.25 vCPU) | 1× (0.25 vCPU) | 1× (0.25 vCPU) | t3.medium (50 GB) | db.t4g.micro | cache.t4g.micro | | `medium` | \$255 | 2× (0.5 vCPU) | 2× (0.25 vCPU) | 1× (0.5 vCPU) | t3.large (100 GB) | db.t4g.small | cache.t4g.micro | | `large` | \$600 | 3× (1 vCPU) | 3× (0.5 vCPU) | 2× (1 vCPU) | r6i.xlarge (500 GB) | db.r6g.large | cache.r6g.large | **Important: Presets vs. individual variables** Presets and individual resource variables (`api_cpu`, `db_instance_class`, etc.) are **mutually exclusive**: * When `sizing = "small|medium|large"`: all individual resource variables are **ignored**. The preset values are used unconditionally. * When `sizing = "custom"`: individual variables take effect normally. If you set `sizing = "medium"` and also `api_cpu = 1024`, the medium preset wins — `api_cpu` has no effect. To override individual values, set `sizing = "custom"` first. ### Custom sizing Set `sizing = "custom"` to control each resource independently: ```hcl theme={null} sizing = "custom" api_cpu = 512 # 0.5 vCPU; raise to 1024+ for sustained >100 req/s api_memory = 1024 api_desired_count = 2 api_autoscale_max = 10 web_desired_count = 2 worker_desired_count = 1 # Data tier (ClickHouse) data_instance_type = "t3.large" # 8 GB - minimum viable for ClickHouse data_volume_size_gb = 100 observability_stack = "none" # none | prometheus | grafana db_instance_class = "db.t4g.small" redis_node_type = "cache.t4g.micro" ``` For high-throughput installs (>100 trace events/sec sustained), bump `data_instance_type` to `m6i.xlarge` and `db_instance_class` to `db.m6g.large`. ### Application options ```hcl theme={null} deployment_mode = "enterprise" # SSO-only login data_retention_days = 90 # ClickHouse TTL; 0 to disable log_retention_days = 30 # CloudWatch log group retention image_tag = "v1.4.0" # specific Observal release; "latest" pulls main ``` See [Configuration](/self-hosting/configuration) for the meaning of each application setting. ### Bring Your Own VPC Deploy Observal into an existing VPC rather than having Terraform create a new one. Use this when you have Transit Gateway, shared-services VPCs, compliance requirements, or peered networks. ```hcl theme={null} # Use an existing VPC vpc_id = "vpc-0abc1234def56789a" private_subnet_ids = ["subnet-aaa11111", "subnet-bbb22222"] public_subnet_ids = ["subnet-xxx11111", "subnet-yyy22222"] # Control ALB placement independently alb_scheme = "internet-facing" # or "internal" ``` When `vpc_id` is set, Terraform skips creating VPC, subnets, IGW, NAT gateway, route tables, and VPC flow logs. All other resources (ALB, ECS, RDS, Redis, data host) are created inside your existing VPC. **VPC requirements:** * DNS support enabled (`enableDnsSupport = true`, `enableDnsHostnames = true`) * Private subnets must have outbound internet access (via NAT Gateway or Transit Gateway) for container image pulls (`ghcr.io`) and AWS API endpoints (SSM, CloudWatch, ECR) * Public subnets must have an Internet Gateway route (only needed if `alb_scheme = "internet-facing"`) * Subnets should span at least 2 AZs for RDS Multi-AZ and ECS placement **Optional: Bring your own security groups:** For environments with strict firewall policies, supply pre-created SG IDs: ```hcl theme={null} alb_security_group_id = "sg-0abc1234def56789a" ecs_security_group_id = "sg-0abc1234def56789b" ``` The ALB SG must allow inbound TCP 80/443 from your desired CIDRs. The ECS SG must allow inbound TCP 8000 and 3000 from the ALB SG, with outbound to all. A full working example lives at [`infra/terraform/aws/examples/byovpc`](https://github.com/Observal/Observal/blob/main/infra/terraform/aws/examples/byovpc/README.md). ### Git webhooks on a private install [MCP repository sync](/use-cases/mcp-repo-sync) publishes a new MCP version as soon as GitHub or GitLab sends a webhook. With `alb_scheme = "internal"`, github.com and gitlab.com cannot reach Observal, so nothing syncs. Turn on the webhook endpoint to fix that without opening the rest of the install: ```hcl theme={null} alb_scheme = "internal" enable_webhook_ingress = true webhook_providers = ["github", "gitlab"] # default ["github"] ``` Terraform then adds an API Gateway HTTP API in front of the existing ALB: * One route per provider, `POST /api/v1/webhooks//mcp/{sync_id}`. Everything else, including other methods, other paths and `..` segments, gets `404` from API Gateway and never reaches the VPC, so the UI, API and login stay private. * A VPC link in the private subnets that connects API Gateway to the ALB's listener, so there are no public subnets, internet gateway or second load balancer. With `enable_tls`, API Gateway connects over HTTPS and checks the ALB certificate against `domain_name`. * A security group rule that lets the VPC link, and nothing else new, reach the ALB. * Throttling (`webhook_throttle_rate_limit`, `webhook_throttle_burst_limit`), access logs in CloudWatch, and `WEBHOOK_PUBLIC_URL` on the API so the MCP **Sync** tab shows the public webhook URL. The webhook URL is the API's `https://.execute-api..amazonaws.com` address, shown by the `webhook_public_url` output. To use your own hostname instead, set `webhook_domain_name` (and `webhook_route53_zone_id` if the public zone differs from `route53_zone_id`); Terraform adds the certificate, custom domain and DNS record, and turns off the `execute-api` address. The receiver rejects any delivery that is not authenticated for that listing (a GitHub HMAC signature or the GitLab secret token), and limits each webhook URL to 60 deliveries a minute. Things to know: * The ALB must be internal. API Gateway connects to the addresses the ALB's DNS name resolves to, and an internet-facing ALB resolves to public addresses the VPC link cannot reach; Terraform stops with an error in that case. An internet-facing ALB that is open to the internet does not need this. * There is no source IP allowlist. GitHub and GitLab do not sign their source addresses, and the providers' ranges change, while every delivery is already authenticated. If you need one, put a REST API with a resource policy, or AWS WAF, in front instead. * A VPC link with no traffic for 60 days goes inactive, and the next delivery fails while AWS reprovisions it, which takes a few minutes. GitHub does not retry failed deliveries automatically; use **Redeliver** on the hook, or run the sync by hand from the **Sync** tab. * API Gateway throttling is approximate and meant as a backstop. Requests over the limit get `429` and show as failed in the provider's delivery log. * VPC links are not available in every Availability Zone (for example `use1-az3`). If `terraform apply` reports an unsupported zone, use private subnets in other zones. * Turning it on adds `WEBHOOK_PUBLIC_URL` to the API task definition, which rolls the API tasks once. * Cost: about \$1 per million deliveries; the VPC link itself is free. ## Required IAM permissions The IAM principal running Terraform needs permission to manage resources across these services. The simplest path is to attach the AWS-managed policies below; for a tighter custom policy, see [Hardened IAM policy](/self-hosting/aws-terraform#hardened-iam-policy). | Service | Managed policy | | - | - | | VPC, EC2 | `AmazonEC2FullAccess` | | ECS | `AmazonECS_FullAccess` | | RDS | `AmazonRDSFullAccess` | | ElastiCache | `AmazonElastiCacheFullAccess` | | Load balancer | `ElasticLoadBalancingFullAccess` | | Certificates | `AWSCertificateManagerFullAccess` | | DNS | `AmazonRoute53FullAccess` | | IAM (creates ECS + EC2 + RDS roles) | `IAMFullAccess` | | Parameter Store + Session Manager | `AmazonSSMFullAccess` | | S3 (backups bucket) | `AmazonS3FullAccess` | | Logs | `CloudWatchLogsFullAccess` | | Application Auto Scaling | `AutoScalingFullAccess` | ## Operating the install ### Tail an ECS service ```bash theme={null} aws logs tail /aws/ecs/observal-prod/api --follow ``` Or get all log group names: ```bash theme={null} terraform output log_group_names ``` ### Force a rolling deploy ```bash theme={null} aws ecs update-service \ --cluster $(terraform output -raw ecs_cluster_name) \ --service observal-prod-api \ --force-new-deployment ``` ### Re-run migrations / seeds The init task runs automatically on every `image_tag` change. To run it manually: ```bash theme={null} $(terraform output -raw init_run_task_command) ``` ### Shell into the data host ```bash theme={null} $(terraform output -raw data_host_ssm_session_command) # inside the session: sudo journalctl -u observal-bootstrap -f # cloud-init / first-boot logs sudo docker compose -f /opt/observal/docker-compose.data.yml ps # CH/Grafana/Prom status sudo docker compose -f /opt/observal/docker-compose.data.yml logs -f clickhouse ``` ### Read a generated secret ```bash theme={null} aws ssm get-parameter --with-decryption \ --name /observal-prod/SECRET_KEY \ --query Parameter.Value --output text ``` The Terraform run generates and stores `DB_PASSWORD`, `CLICKHOUSE_PASSWORD`, `SECRET_KEY`, and `GRAFANA_ADMIN_PASSWORD`, plus pre-built `DATABASE_URL`, `REDIS_URL`, and `CLICKHOUSE_URL` connection strings. ECS injects these into tasks at start; you never paste them in. ### Upgrade to a new Observal release ```hcl theme={null} # terraform.tfvars image_tag = "v1.5.0" ``` ```bash theme={null} terraform apply ``` The init task re-runs migrations against the live RDS, then ECS rolls each service over with zero downtime. ### Resize the data volume ```hcl theme={null} # terraform.tfvars data_volume_size_gb = 250 ``` ```bash theme={null} terraform apply # then SSM into the data host: sudo growpart /dev/nvme1n1 1 || true sudo resize2fs /dev/nvme1n1 ``` ### Destroy ```bash theme={null} terraform destroy ``` In `prod`, RDS has `deletion_protection = true` and `skip_final_snapshot = false`. Disable both manually before destroy if that is really what you want. The S3 backups bucket also blocks destroy unless `backup_bucket_force_destroy = true`. ## Remote state By default, Terraform writes state to your laptop (`terraform.tfstate`). For a real install, configure remote state so the install can be managed by anyone on your team and state isn't lost. 1. Create an S3 bucket (versioned, encrypted) and a DynamoDB table with hash key `LockID`. 2. Uncomment and fill the backend block in `versions.tf`: ```hcl theme={null} backend "s3" { bucket = "your-tf-state-bucket" key = "observal/prod/terraform.tfstate" region = "us-east-1" dynamodb_table = "your-tf-lock-table" encrypt = true } ``` 3. Re-run `terraform init` and answer "yes" when prompted to migrate state. ## Cost expectations Rough monthly baseline in `us-east-1` at on-demand rates (May 2026): | Component | \~\$/month | | - | - | | Fargate api 2× (0.5 vCPU / 1 GB) | \$30 | | Fargate web 2× (0.25 vCPU / 0.5 GB) | \$15 | | Fargate worker 1× (0.5 vCPU / 1 GB) | \$15 | | EC2 `t3.large` (data host) | \$60 | | RDS `db.t4g.small` Multi-AZ | \$50 | | ElastiCache (2× `cache.t4g.micro`) | \$25 | | ALB | \$20 | | Webhook API Gateway (only with `enable_webhook_ingress`) | \<\$1 | | NAT Gateway | \$33 + egress | | EBS gp3 100 GB | \$8 | | S3 backups (1 GB cold) | \$0.10 | | **Baseline** | **\~\$255** | Set `environment = "staging"` to drop RDS to single-AZ, run a single Redis node, and skip RDS deletion protection, typically halves the bill. Drop `worker_desired_count` and `web_desired_count` for further savings. ## Production hardening checklist The defaults are safe but conservative. Before pointing real traffic at this: * [ ] Switch to remote state (S3 + DynamoDB) * [ ] Restrict `alb_ingress_cidrs` to known networks * [ ] Enable AWS GuardDuty + Config in the account * [ ] Add CloudWatch alarms on RDS `CPUUtilization`, `FreeableMemory`, ECS service CPU, ALB `HTTPCode_Target_5XX_Count` * [ ] Attach AWS WAF to the ALB * [ ] Set `transit_encryption_enabled = true` on the ElastiCache replication group and switch `REDIS_URL` to `rediss://...` * [ ] Configure Observal SSO. See [Authentication and SSO](/self-hosting/authentication) * [ ] Test the [backup and restore](/self-hosting/backup-and-restore) procedure end-to-end * [ ] Replace the GitHub tarball download in `user-data.sh.tftpl` with an artifact URL you control ## Troubleshooting **`terraform apply` succeeds but the URL returns 502.** The api/web tasks are still starting. Watch the service: ```bash theme={null} aws ecs describe-services \ --cluster $(terraform output -raw ecs_cluster_name) \ --services observal-prod-api \ --query 'services[0].deployments' aws logs tail /aws/ecs/observal-prod/api --follow ``` The `null_resource.run_init` step also runs migrations before the api service comes up. If the init task fails, the api service won't start. **Init task fails.** Check the `/aws/ecs/observal-prod/init` log group. The most common failures are: * RDS not yet reachable (transient on first apply; re-running fixes it) * Migration error (look at the entrypoint output) To re-run by hand: `$(terraform output -raw init_run_task_command)`. **ALB target health is `unhealthy`.** * For `api`: health check hits `/readyz` on port 8000. Tail the api log group for startup errors. * For `web`: health check hits `/` on port 3000. Check the web log group. * For Grafana: health check hits `/api/health` on port 3001. SSM into the data host and check `docker compose ps`. **ACM certificate stuck in `Pending validation`.** The DNS validation records were not created in your hosted zone. Verify `route53_zone_id` is correct and that the IAM principal had `route53:ChangeResourceRecordSets` permission at apply time. Re-run `terraform apply` after fixing. **RDS storage is full.** `max_allocated_storage` autoscales up to 500 GB by default. Raise `db_max_allocated_storage_gb` if you have heavier audit-log volume. For application-level issues (login fails, traces missing) see [Troubleshooting](/self-hosting/troubleshooting). # Backup and restore Source: https://docs.observal.io/self-hosting/backup-and-restore What to back up, how often, and how to restore. Observal has three persistent stores with very different loss profiles. ## What matters, in order | Priority | Volume | Contents | Loss impact | | - | - | - | - | | **Critical** | `apidata` | JWT signing keys | Every session invalidated. No recovery without the keys. | | **Critical** | `pgdata` | Users, RBAC, registry metadata, agents | All accounts and registry lost. | | **Important** | `chdata` | Session events, aggregates, audit, and security events | All telemetry lost. Accounts and registry survive. | | **Low** | `grafanadata` | Custom Grafana dashboards | Custom dashboards lost; provisioned defaults come back automatically. | | **Low** | `redisdata` | Job queue state | In-flight jobs lost; they re-queue on next worker restart. | **Always back up `apidata` and `pgdata` together.** Backing up one without the other leaves you in a broken state after restore. ## Backup cadence | Cadence | What | Retention | | - | - | - | | Daily | `pgdata`, `apidata` | 30 days | | Weekly | `chdata` | 12 weeks | | Before every upgrade | All three | Keep until the upgrade is confirmed stable | ## Postgres backup Use `pg_dump` inside the running container: ```bash theme={null} docker compose -f docker/docker-compose.yml exec -T observal-db \ pg_dump -U postgres observal | gzip > observal-pg-$(date +%Y%m%d).sql.gz ``` Restore into a fresh DB: ```bash theme={null} docker compose -f docker/docker-compose.yml down -v docker compose -f docker/docker-compose.yml up -d observal-db zcat observal-pg-20260421.sql.gz | \ docker compose -f docker/docker-compose.yml exec -T observal-db \ psql -U postgres observal docker compose -f docker/docker-compose.yml up -d ``` ## JWT key backup (`apidata` volume) The API container mounts the `apidata` volume at `/data` and stores keys at `/data/keys`. Tar it out: ```bash theme={null} docker compose -f docker/docker-compose.yml exec -T observal-api \ tar czf - -C /data keys > observal-keys-$(date +%Y%m%d).tar.gz ``` Restore: ```bash theme={null} # With the API stopped docker compose -f docker/docker-compose.yml stop observal-api docker run --rm \ -v observal_apidata:/data \ -v "$(pwd)":/backup \ alpine sh -c "cd /data && tar xzf /backup/observal-keys-20260421.tar.gz" docker compose -f docker/docker-compose.yml start observal-api ``` ## ClickHouse backup ### Option A - volume snapshot (simplest) ```bash theme={null} docker compose -f docker/docker-compose.yml stop observal-clickhouse docker run --rm -v observal_chdata:/data -v "$(pwd)":/backup \ alpine tar czf /backup/observal-ch-$(date +%Y%m%d).tar.gz -C /data . docker compose -f docker/docker-compose.yml start observal-clickhouse ``` Downtime: however long the tar takes (a minute to tens of minutes depending on size). ### Option B - ClickHouse native `BACKUP` (no downtime) ```bash theme={null} docker compose -f docker/docker-compose.yml exec observal-clickhouse \ clickhouse-client --query "BACKUP DATABASE observal TO Disk('backups', 'observal-$(date +%Y%m%d).zip')" ``` Requires configuring a backup disk in ClickHouse config; see [ClickHouse docs](https://clickhouse.com/docs/en/operations/backup). Restore: ```bash theme={null} docker compose -f docker/docker-compose.yml exec observal-clickhouse \ clickhouse-client --query "RESTORE DATABASE observal FROM Disk('backups', 'observal-20260421.zip')" ``` ## Restore order If you're restoring from backup after a catastrophic failure: 1. Stop the whole stack: `docker compose down`. 2. Restore `apidata` (JWT keys) first. 3. Restore `pgdata` (Postgres). 4. Restore `chdata` (ClickHouse). 5. Bring up the stack: `docker compose up -d`. 6. Smoke test: `observal auth login`, `observal auth status`. Skipping step 2 works but every user has to re-login. ## Verifying a backup Test restores in a staging environment at least quarterly. Untested backups are guesses. Smoke test after restore: ```bash theme={null} observal auth login observal auth whoami # you should be your pre-backup user observal agent list # registry should be intact observal ops traces --limit 5 # traces up to the backup timestamp should be visible ``` ## Automated backup A minimal cron setup (on the Docker host): ```cron theme={null} # Daily at 03:00 - Postgres + JWT keys 0 3 * * * cd /opt/Observal && \ docker compose -f docker/docker-compose.yml exec -T observal-db \ pg_dump -U postgres observal | gzip > /backups/pg-$(date +\%Y\%m\%d).sql.gz 0 3 * * * cd /opt/Observal && \ docker compose -f docker/docker-compose.yml exec -T observal-api \ tar czf - -C /data keys > /backups/keys-$(date +\%Y\%m\%d).tar.gz # Weekly Sunday at 04:00 - ClickHouse 0 4 * * 0 cd /opt/Observal && \ docker compose -f docker/docker-compose.yml stop observal-clickhouse && \ docker run --rm -v observal_chdata:/data -v /backups:/backup alpine \ tar czf /backup/ch-$(date +\%Y\%m\%d).tar.gz -C /data . && \ docker compose -f docker/docker-compose.yml start observal-clickhouse ``` Ship the `/backups` directory offsite (S3, B2, rsync to another host). ## Next → [Troubleshooting](/self-hosting/troubleshooting) # Configuration Source: https://docs.observal.io/self-hosting/configuration Boot-time infrastructure settings live in `.env`. Credentials can use dedicated files. Runtime settings, including SSO, normally live in the admin UI, while file-backed sensitive settings remain only in process memory. ## Required for production Source deployments must override these before going live. Server-package setup generates the secret values and file references automatically: | Variable | Default | Why change | | - | - | - | | `SECRET_KEY` or `SECRET_KEY_FILE` | `change-me-to-a-random-string` | Application encryption secret. Use a random value of at least 32 characters. | | `POSTGRES_PASSWORD` or `POSTGRES_PASSWORD_FILE` | `postgres` | PostgreSQL bootstrap credential. | | `CLICKHOUSE_PASSWORD` or generated hashed user config | `clickhouse` | ClickHouse credential. | | `CORS_ALLOWED_ORIGINS` | `http://localhost:3000` | Scope to your real frontend origin(s). Configure as `deployment.cors_origins` in Admin Settings. | | `deployment.frontend_url` | `http://localhost:3000` | Used for OAuth redirects and email links. Configure in Admin Settings. | The server-package installer generates these credentials in an operator-owned `secrets/` directory. Directories use mode `0750`, files use mode `0640`, and `OBSERVAL_SECRET_GID` grants the containers read access through the operator's group. Existing deployments can keep direct environment values. With a terminal, setup prompts for the frontend URL, bind address, and observability profile. Without a terminal, the same installer selects `http://localhost:3000`, loopback binding, and no optional observability stack. It prints the initial super-admin email, generated password, and password-file path after the first successful startup. Existing headless installations preserve their configuration rather than replacing it. ## Secret files For any supported credential named `NAME`, set `NAME_FILE` to a UTF-8 file path instead of placing the value in `.env`: ```dotenv theme={null} SECRET_KEY_FILE=/run/secrets/secret_key DATABASE_URL_FILE=/run/secrets/database_url JWT_KEY_PASSWORD_FILE=/run/secrets/jwt_key_password GIT_CLONE_TOKEN_FILE=/run/secrets/git_clone_token OAUTH_CLIENT_SECRET_FILE=/run/secrets/oauth_client_secret INSIGHTS_API_KEY_FILE=/run/secrets/insights_api_key ``` The reader accepts files up to 64 KiB and removes one trailing newline. Setting both `NAME` and `NAME_FILE` is an error, so precedence is never ambiguous. Replace a file atomically and restart the affected API or worker processes to rotate the value without rebuilding Observal. File-backed dynamic settings are marked externally managed in the admin API. Their contents are not imported into PostgreSQL or Redis, and the admin API rejects attempts to overwrite, delete, or revoke them. Supported file-backed boot credentials include `DATABASE_URL`, `CLICKHOUSE_URL`, `REDIS_URL`, `SECRET_KEY`, `OLD_SECRET_KEY`, `JWT_KEY_PASSWORD`, `GIT_CLONE_TOKEN`, and every `DEMO_*_PASSWORD`. SSO environment imports support the same form, including OAuth client secrets, Google and GitHub OAuth secrets, the insights provider key, SAML certificates, and the SAML key-encryption password. CLI tokens support `OBSERVAL_ACCESS_TOKEN_FILE`, `OBSERVAL_API_KEY_FILE`, and `OBSERVAL_TOKEN_FILE`. PostgreSQL containers use `POSTGRES_PASSWORD_FILE`. The server package generates a hashed ClickHouse user configuration instead of putting its database password in `.env`. Service-specific subdirectories expose only the PostgreSQL password to PostgreSQL, the ClickHouse health credential to ClickHouse, and the Grafana and ClickHouse datasource credentials to Grafana. ## SSO-only mode Set `deployment.sso_only=true` in **Admin → SSO** when you want IdP-only access. Leave it `false` to keep password login available. ## Demo accounts Seeded on first startup *only* when no users exist: ```dotenv theme={null} DEMO_SUPER_ADMIN_EMAIL=super@demo.example DEMO_SUPER_ADMIN_PASSWORD=super-changeme DEMO_ADMIN_EMAIL=admin@demo.example DEMO_ADMIN_PASSWORD=admin-changeme DEMO_REVIEWER_EMAIL=reviewer@demo.example DEMO_REVIEWER_PASSWORD=reviewer-changeme DEMO_USER_EMAIL=user@demo.example DEMO_USER_PASSWORD=user-changeme ``` **Remove demo account variables and their password files before a real deployment.** Existing demo users survive after removal. Delete them manually (`observal admin delete-user `). > **Admin settings warning:** If demo accounts are still active or `SECRET_KEY` is insecure, the admin Settings page will display a warning banner at the top so operators can spot and fix the issue without digging through logs. ## Database connections Source deployments may use direct values: ```dotenv theme={null} DATABASE_URL=postgresql+asyncpg://postgres:postgres@observal-db:5432/observal CLICKHOUSE_URL=clickhouse://default:clickhouse@observal-clickhouse:8123/observal REDIS_URL=redis://observal-redis:6379 ``` Server-package installs use file references instead: ```dotenv theme={null} DATABASE_URL_FILE=/run/secrets/database_url CLICKHOUSE_URL_FILE=/run/secrets/clickhouse_url REDIS_URL_FILE=/run/secrets/redis_url ``` Inside Docker Compose, hostnames resolve via the `observal-net` bridge (e.g. `observal-db`). Outside Docker (e.g. CLI running on host against dockerized DBs), use `localhost:`. ## OAuth / SSO Optional. Configure OIDC, SAML, and SSO-only mode in **Admin → SSO**. OIDC client changes are stored immediately, then take effect after the API restarts. Full setup in [Authentication and SSO](/self-hosting/authentication). ## Rate limiting ``` RATE_LIMIT_AUTH=10/minute # general auth endpoints RATE_LIMIT_AUTH_STRICT=5/minute # user registration ``` Tighten for higher-traffic deployments. ## ClickHouse retention ``` DATA_RETENTION_DAYS=90 ``` Session events older than this are removed from ClickHouse. Set to `0` to disable retention. The minimum non-zero value enforced on startup is 7. ## JWT keys ``` JWT_SIGNING_ALGORITHM=ES256 # ES256 (default) or RS256 JWT_KEY_DIR=/data/keys # persisted in the apidata volume ``` The server generates asymmetric keys on first boot and stores them in `$JWT_KEY_DIR`. **Back up this directory**: losing the keys invalidates every session. Changing `JWT_SIGNING_ALGORITHM` and restarting retires the current public key, generates the selected key type, and keeps old tokens verifiable during their normal lifetime. SAML service-provider material can be mounted independently: ```dotenv theme={null} SAML_SP_PRIVATE_KEY_FILE=/run/secrets/saml_sp_private_key SAML_SP_X509_CERT_FILE=/run/secrets/saml_sp_x509_cert ``` Set both files together. They override database-generated SP material, are reported as externally managed, and cannot be regenerated or removed through the admin API. Replace both files and restart the API to rotate them. More: [Authentication and SSO](/self-hosting/authentication). ## Git operations (submission analysis) ```dotenv theme={null} ALLOW_INTERNAL_GIT_URLS=false # allow internal/private Git URLs (GitLab/GHE) GIT_CLONE_TOKEN_FILE=/run/secrets/git_clone_token GIT_CLONE_TOKEN_HOSTS=github.com # exact HTTPS hosts permitted to receive the token GIT_CLONE_TOKEN_USER=x-access-token GIT_CLONE_TIMEOUT=120 # seconds ``` `GIT_CLONE_TOKEN_HOSTS` is required for authenticated clones: without it the token is never sent. List exact hostnames separated by commas (for example `github.com,gitlab.example.com`, or `git.example.com:8443` for a nonstandard port); subdomains and lookalikes are not trusted automatically. Authorities match exactly: `github.com:443` requires its own entry and does not match `github.com`. Only HTTPS clones receive credentials, even if HTTP clones are enabled. `GIT_CLONE_TOKEN_USER` varies by provider: `x-access-token` for GitHub, `oauth2` or `private-token` for GitLab. ## Observal CLI (client-side) env vars Not set in `.env` on the server. These live on the CLI user's machine. | Variable | Purpose | | - | - | | `OBSERVAL_SERVER_URL` | Default server URL | | `OBSERVAL_ACCESS_TOKEN` / `OBSERVAL_API_KEY` | Pre-authenticate without `login` | | `OBSERVAL_TIMEOUT` | Request timeout (seconds) | Full list: [Environment variables](/reference/environment-variables). ## Next → [Ports and volumes](/self-hosting/ports-and-volumes) # Databases Source: https://docs.observal.io/self-hosting/databases Observal runs two DBs with very different jobs. | DB | Role | Access pattern | Schema source of truth | | - | - | - | - | | Postgres 16 | Registry, users, config | Relational, transactional | Alembic migrations in `observal-server/alembic/versions/` | | ClickHouse 26.5 | Telemetry and audit event storage | Columnar, time-series, high-write | Versioned SQL migrations in `observal-server/clickhouse/migrations/` | ## Postgres ### What's in it * `users`, `roles`, RBAC bindings * `mcps`, `agents`, `skills`, `hooks`, `prompts`, `sandboxes`: registry metadata * `reviews`: submission review state * `feedback`, `ratings` * `alerts`, `alert_history` * `api_keys` * `audit_log` and related audit tables ### Migrations Managed by Alembic. The server applies pending migrations automatically on startup. Migration files live in `observal-server/alembic/versions/`. For Docker Compose deployments, run the init service manually when needed: ```bash theme={null} docker compose -f docker/docker-compose.yml run --rm observal-init ``` The init service applies Alembic and ClickHouse migrations before API startup. `observal server migrate` moves data between deployments; it does not apply schema migrations. ### Reset To wipe the registry and start over: ```bash theme={null} docker compose -f docker/docker-compose.yml down -v docker compose -f docker/docker-compose.yml up --build -d ``` The `-v` deletes all named volumes. Use only in dev. *** ## ClickHouse ### What's in it Core tables: | Table | Contents | | - | - | | `session_events` | Raw and parsed harness JSONL lines, token fields, tool fields, and session metadata | | `session_stats_agg` | Pre-aggregated session list and summary metrics from `session_events` | | `layer_snapshots` | Harness config snapshots used by version-aware insights | | `audit_log` | Audit events | | `security_events` | Security events for login, auth, and admin activity | | `webhook_deliveries` | Alert webhook delivery attempts and status | ### Deduplication and aggregates `session_events` and `layer_snapshots` use `ReplacingMergeTree` for idempotent ingest. `session_stats_agg` uses `AggregatingMergeTree` and is maintained by a materialized view. The API query layer handles the required `FINAL` or aggregate reads. If you query ClickHouse directly, match the table engine instead of assuming every table reads the same way. ### Retention (TTL) Controlled by `DATA_RETENTION_DAYS`: * Default `90`: rows older than 90 days are TTL'd out. * `0`: retention disabled (disk grows without bound). * The server enforces a minimum of `7` on any non-zero value. TTL runs asynchronously. Disk space is reclaimed on the next merge; don't expect instant free-up. ### Schema migrations ClickHouse schema changes are managed separately from Alembic. Alembic is only for Postgres. ClickHouse migration files live in: ```bash theme={null} observal-server/clickhouse/migrations/*.sql ``` The init container runs ClickHouse migrations after Alembic and before the API starts. The migration runner records applied files in `clickhouse_schema_migrations`. On existing installations that predate versioned ClickHouse migrations, the runner detects the existing baseline tables and stamps `001_baseline.sql` as applied instead of replaying the whole baseline. For local checks outside Docker, run the same runner from the server package: ```bash theme={null} cd observal-server python -m services.clickhouse.migrations ``` Do not put ClickHouse DDL in startup code. Add a new migration file instead. ### Capacity planning Session record size depends on harness transcript detail and tool output size. Measure representative sessions, apply the configured raw-line retention window, and plan 2 to 3 times headroom for merges and replicas. ### External ClickHouse For heavy workloads, run ClickHouse on a dedicated host outside the compose stack. Point the API at it: ``` CLICKHOUSE_URL=clickhouse://user:pass@external-clickhouse.example.com:8123/observal ``` Remove the `observal-clickhouse` service from `docker-compose.yml` or ignore it. *** ## Backup See [Backup and restore](/self-hosting/backup-and-restore). Short version: * Postgres: `pg_dump` from a running container. * ClickHouse: snapshot the `chdata` volume, or use ClickHouse's native `BACKUP` command. * Both: back up before every upgrade. ## Next → [Authentication and SSO](/self-hosting/authentication) # Docker Compose setup Source: https://docs.observal.io/self-hosting/docker-compose Step-by-step bring-up of the Observal stack. End state: the core services are healthy, API responding at `http://localhost/health`, web UI at `http://localhost`. Prometheus and Grafana are optional. ## 1. Clone and configure ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal cp .env.example .env ``` The `.env.example` ships with working direct-value defaults for local source development, including demo account credentials. You do not need to edit it for local development. Production server-package installs use generated files under `secrets/` instead; see [Configuration](/self-hosting/configuration#secret-files). > [!NOTE] > > You need Docker Engine ≥ 24.0 with Compose v2 (`docker compose`, not `docker-compose`). Homebrew's Docker formula is outdated. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages. Verify with `docker version` and `docker compose version`. ## 2. Start the stack Core stack only: ```bash theme={null} docker compose -f docker/docker-compose.yml up --build -d ``` With Prometheus only: ```bash theme={null} docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up --build -d ``` With Prometheus and Grafana: ```bash theme={null} COMPOSE_PROFILES=grafana docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up --build -d ``` First build takes a few minutes (pulls images, builds `observal-api` and `observal-web`). Subsequent starts are fast. ## 3. Verify health ```bash theme={null} docker compose -f docker/docker-compose.yml ps ``` Every service should show `healthy` or `running`. The API waits for Postgres, ClickHouse, and Redis to pass health checks before starting. Expect 15–30 seconds on first boot. Hit the health endpoint: ```bash theme={null} curl http://localhost/health # {"status":"ok"} ``` ## 4. Configure TLS (production only) For local dev, `http://localhost` is fine. For production, put a TLS-terminating reverse proxy in front of the nginx LB. See [Requirements → TLS / HTTPS](/self-hosting/requirements#tls--https). ## 5. Bootstrap the first user ### Option A - demo accounts (fastest for trying it out) `.env.example` seeds four demo accounts on first startup: | Role | Email | Password | | - | - | - | | Super Admin | `super@demo.example` | `super-changeme` | | Admin | `admin@demo.example` | `admin-changeme` | | Reviewer | `reviewer@demo.example` | `reviewer-changeme` | | User | `user@demo.example` | `user-changeme` | Log in with the CLI: ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash # if you haven't already observal auth login # Email: super@demo.example, Password: super-changeme ``` **Remove demo accounts before real deployment.** Unset the `DEMO_*` env vars in `.env` and restart. Already-seeded accounts stay until you delete them manually (`observal admin delete-user `). ### Option B - fresh bootstrap (recommended for production) Remove `DEMO_*` from `.env` and start the stack. Run: ```bash theme={null} observal auth login # Server URL: http://localhost # No users detected - bootstrapping admin account. # Email: alice@your-company.com # Password: ************** ``` The CLI detects that no users exist and interactively creates the first admin. The `/api/v1/auth/bootstrap` endpoint is restricted to localhost access for security. ## 6. Verify with the CLI ```bash theme={null} observal auth whoami observal auth status observal registry mcp list # empty list - you haven't added anything yet ``` ## 7. Stop, restart, rebuild ```bash theme={null} # Stop core and any optional monitoring containers make down # Stop and delete all data, including optional monitoring volumes docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml --profile grafana down -v # Restart one service docker compose -f docker/docker-compose.yml restart observal-api # Rebuild after code changes docker compose -f docker/docker-compose.yml up --build -d observal-api ``` Makefile shortcuts from the repo root: ```bash theme={null} make logs # tail core service logs make rebuild # rebuild and restart core services make up-prometheus # start core services with Prometheus make up-observability # start core services with Prometheus and Grafana make rebuild-prometheus # rebuild core services with Prometheus make rebuild-observability # rebuild core services with Prometheus and Grafana ``` ## 8. Logs ```bash theme={null} docker compose -f docker/docker-compose.yml logs -f # all docker compose -f docker/docker-compose.yml logs -f observal-api # one service ``` ## 9. Port conflicts If `docker compose up` fails with `port is already allocated`, remap host ports via env vars: ```bash theme={null} POSTGRES_HOST_PORT=5433 REDIS_HOST_PORT=6380 \ docker compose -f docker/docker-compose.yml up --build -d ``` Every host port is configurable. See [Ports and volumes](/self-hosting/ports-and-volumes) for the full list. ## Next → [Configuration](/self-hosting/configuration): which env vars to change for production. # GCP deployment with Terraform Source: https://docs.observal.io/self-hosting/gcp-terraform End state: Observal running in your GCP project on Cloud Run, backed by Cloud SQL Postgres, Memorystore Redis, and a GCE instance for ClickHouse — with a Global HTTPS Load Balancer, managed SSL certificate, Secret Manager for credentials, and GCS for backups. For the overall deployment strategy and comparison with AWS, see [Production deployment](/self-hosting/production-deploy). If you want a simpler single-VM setup, see [Single-node deployment](/self-hosting/single-node-deploy). ## What gets provisioned A single `terraform apply` creates: * **VPC** with a private subnet, Cloud NAT, and a Serverless VPC Access Connector * **Cloud Run v2 services**: `api`, `web`, `worker` with autoscaling (min/max instance counts) * **Cloud Run v2 job**: `init` (one-shot migrations + seeds) * **Cloud SQL Postgres** (Open-source distribution): optional HA, encrypted, automated backups * **Memorystore Redis**: BASIC or STANDARD\_HA tier * **GCE instance** (data host): ClickHouse on a persistent disk, with optional Prometheus and Grafana, accessible via IAP SSH tunnel * **Global HTTPS Load Balancer** with managed SSL certificate (when domain is supplied) * **Cloud DNS** A record pointing to the load balancer * **Secret Manager**: generated DB / ClickHouse / SECRET\_KEY passwords, plus connection URLs injected into Cloud Run services * **GCS backups bucket**: versioned, lifecycle to Nearline → delete * **Artifact Registry**: for storing container images (optional) ## Prerequisites | Requirement | Why | | - | - | | GCP project with billing enabled | Resources cost money | | `gcloud` CLI authenticated | `gcloud auth application-default login` | | Terraform ≥ 1.5 | `brew install terraform` | | (Optional) Cloud DNS managed zone | Required for HTTPS on a custom domain | Enable required APIs: ```bash theme={null} gcloud services enable \ run.googleapis.com \ sqladmin.googleapis.com \ redis.googleapis.com \ compute.googleapis.com \ secretmanager.googleapis.com \ dns.googleapis.com \ vpcaccess.googleapis.com \ servicenetworking.googleapis.com \ iam.googleapis.com ``` ## Quickstart ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal/infra/terraform/gcp # 1. Authenticate gcloud auth application-default login # 2. Configure cp terraform.tfvars.example terraform.tfvars $EDITOR terraform.tfvars # At minimum, set project_id # 3. Apply terraform init terraform plan -out tf.plan terraform apply tf.plan # 4. Run migrations $(terraform output -raw init_job_run_command) # 5. Verify terraform output app_url ``` ## Configuration All inputs live in `terraform.tfvars`. ### Minimal (no custom domain) ```hcl theme={null} project_id = "my-gcp-project" region = "us-central1" environment = "prod" ``` The install comes up on the Cloud Run default URL (e.g. `https://observal-prod-api-xxxxx-uc.a.run.app`). ### Recommended (HTTPS on your domain) ```hcl theme={null} project_id = "my-gcp-project" region = "us-central1" environment = "prod" domain_name = "observal.example.com" dns_managed_zone_name = "example-com" ``` Terraform provisions a Global HTTPS Load Balancer with a Google-managed SSL certificate and creates the DNS A record. ### Sizing ```hcl theme={null} # Cloud Run api_cpu = "1" api_memory = "1Gi" api_min_instances = 1 api_max_instances = 10 web_min_instances = 1 worker_min_instances = 1 # Data tier data_machine_type = "e2-standard-2" # 2 vCPU / 8 GB data_disk_size_gb = 100 # Cloud SQL db_tier = "db-g1-small" db_disk_size_gb = 50 db_ha_enabled = false # set true for production HA # Redis redis_memory_size_gb = 1 redis_tier = "BASIC" # or STANDARD_HA ``` For high-throughput installs, bump `data_machine_type` to `e2-standard-4`, `db_tier` to `db-custom-4-16384`, and `redis_tier` to `STANDARD_HA`. ## Operating the install ### Shell into the data host ```bash theme={null} $(terraform output -raw data_host_ssh_command) # Inside: sudo docker compose -f /opt/observal/docker-compose.data.yml ps sudo docker compose -f /opt/observal/docker-compose.data.yml logs -f clickhouse ``` No public SSH. Access is through IAP (Identity-Aware Proxy) — authenticated, audited, no SSH keys to manage. ### View logs Cloud Run logs go to Cloud Logging automatically: ```bash theme={null} gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=observal-prod-api" \ --limit 50 --format json ``` Or use the Cloud Console: **Logging → Logs Explorer**, filter by `resource.type="cloud_run_revision"`. ### Re-run migrations ```bash theme={null} $(terraform output -raw init_job_run_command) ``` ### Read a generated secret ```bash theme={null} gcloud secrets versions access latest \ --secret="observal-prod-SECRET_KEY" ``` ### Upgrade to a new release ```hcl theme={null} # terraform.tfvars image_tag = "v1.5.0" ``` ```bash theme={null} terraform apply $(terraform output -raw init_job_run_command) ``` Cloud Run handles the rolling deploy. Zero downtime. ### Resize the data disk ```hcl theme={null} data_disk_size_gb = 250 ``` ```bash theme={null} terraform apply # SSH into the data host and resize the filesystem: $(terraform output -raw data_host_ssh_command) sudo growpart /dev/sdb 1 || true sudo resize2fs /dev/sdb1 ``` ### Destroy ```bash theme={null} terraform destroy ``` Cloud SQL deletion protection is enabled on prod. Disable manually before destroy if you mean it. ## Cost estimate (us-central1, on-demand) | Component | Spec | \~\$/month | | - | - | - | | Cloud Run api | 1 vCPU / 1 GB, min 1 instance | \$25 | | Cloud Run web | 1 vCPU / 512 MB, min 1 instance | \$15 | | Cloud Run worker | 1 vCPU / 1 GB, min 1 instance | \$25 | | Cloud SQL Postgres | db-g1-small (shared) | \$25 | | Memorystore Redis | 1 GB BASIC | \$35 | | GCE data host | e2-standard-2 | \$50 | | Global HTTPS LB | — | \$18 | | Persistent Disk 100 GB | — | \$4 | | GCS backups | \~1 GB | \$0.02 | | **Total** | | **\~\$200** | Cloud Run scales to zero when idle (if `min_instances = 0`), which can significantly reduce costs for low-traffic deployments. ## Production hardening checklist * [ ] Set `db_ha_enabled = true` for Cloud SQL high availability * [ ] Set `redis_tier = "STANDARD_HA"` for Redis failover * [ ] Restrict Cloud Run ingress (use `ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"` if behind GCLB) * [ ] Enable Cloud Armor (WAF) on the Global HTTPS Load Balancer * [ ] Enable Security Command Center in the project * [ ] Set up alerting on Cloud SQL CPU, memory, and Cloud Run error rates * [ ] Configure [SSO](/self-hosting/authentication) * [ ] Test [backup and restore](/self-hosting/backup-and-restore) end-to-end * [ ] Set up Terraform remote state in GCS ## Troubleshooting **Cloud Run service stuck in `Revision is not ready`.** Check revision logs in Cloud Logging. Common causes: Secret Manager permissions missing, VPC connector not ready, or image pull failure. **Init job fails.** ```bash theme={null} gcloud run jobs executions list --job=observal-prod-init --region=us-central1 gcloud logging read "resource.type=cloud_run_job AND resource.labels.job_name=observal-prod-init" --limit 20 ``` Usually: Cloud SQL not yet reachable (transient on first apply), or missing secret access. **502 from the load balancer.** The managed SSL certificate takes 10–30 minutes to provision. Check: **Network Services → Load Balancing → (your LB) → Backend services → Health**. **Cloud SQL connection refused from Cloud Run.** The VPC Access Connector must be in the same region as Cloud Run. Verify `connector_cidr` doesn't overlap with your VPC subnet. For application-level issues, see [Troubleshooting](/self-hosting/troubleshooting). ## Next * [Production deployment](/self-hosting/production-deploy) — overview of both clouds * [Configuration](/self-hosting/configuration) — environment variables * [Upgrades](/self-hosting/upgrades) — upgrade and rollback procedures * [Backup and restore](/self-hosting/backup-and-restore) — restore procedures # Overview Source: https://docs.observal.io/self-hosting/index Run Observal entirely on your own infrastructure. No SaaS, no egress, every byte of telemetry stays inside your network. For a packaged single-node install, `install-server.sh` generates restricted credential files under `secrets/`, records their `NAME_FILE` paths in `.env`, and binds host ports to loopback. It prompts when a terminal is available and applies the same safe defaults automatically in headless CI or agent environments. The source Compose path remains available for development and accepts direct `.env` values. ## Architecture at a glance ```mermaid theme={null} flowchart TB engineers[Engineers] web["observal-web - static web UI"] api["observal-api - FastAPI backend"] worker["observal-worker - async jobs"] db["observal-db - Postgres"] redis["observal-redis - jobs + pub/sub"] ch["observal-clickhouse - telemetry"] grafana["observal-grafana - optional dashboards"] engineers --> web web <--> api api --> worker api --> db api --> redis api --> ch worker --> db worker --> redis worker --> ch grafana --> ch ``` **Seven services:** | Service | Image | Ports | Purpose | | - | - | - | - | | `observal-api` | built from `docker/Dockerfile.api` | 8000 | FastAPI backend | | `observal-web` | built from `docker/Dockerfile.web` | 3000 | Vite web UI | | `observal-db` | `postgres:16` | 5432 | Registry, users, config | | `observal-clickhouse` | `clickhouse/clickhouse-server:26.3` | 8123 | Session and audit events | | `observal-redis` | `redis:7-alpine` | 6379 | Job queue (arq) + pub/sub | | `observal-worker` | built from `docker/Dockerfile.api` | (internal) | Background async jobs | | `observal-grafana` | `grafana/grafana-oss:11.6.5` | 3001 | Dashboards (optional) | All services run on a private `observal-net` bridge network. Named volumes (`pgdata`, `chdata`, `redisdata`, `grafanadata`, `apidata`) hold persistent data. ## Deployment tiers Choose the deployment model that fits your team: | | [Single-node](/self-hosting/single-node-deploy) | [Kubernetes](https://github.com/Observal/Observal/blob/main/docs/self-hosting/kubernetes-helm.md) | [Production](/self-hosting/production-deploy) | | - | - | - | - | | **How** | Docker Compose on one VM | Official Helm chart | Terraform on AWS or GCP | | **Best for** | ≤50 users, internal tools, POCs | Cloud-native teams, existing K8s infra | Enterprise, SLA-bound, 50+ users | | **Cost** | $20 to $150/mo | Variable | \~$180 to $255/mo | | **HA** | No | Pod resilience & horizontal scaling | Yes (Multi-AZ databases, autoscaling) | | **Time to deploy** | 10 minutes | 10 to 15 minutes | 20 to 30 minutes | **Start here:** | If you want to... | Read | | - | - | | Deploy on a single VM (simplest) | [Single-node deployment](/self-hosting/single-node-deploy) | | Deploy on Kubernetes with Helm | [Kubernetes deployment with Helm](https://github.com/Observal/Observal/blob/main/docs/self-hosting/kubernetes-helm.md) | | Deploy a production HA stack | [Production deployment](/self-hosting/production-deploy) | | Deploy on AWS specifically | [AWS deployment with Terraform](/self-hosting/aws-terraform) | | Deploy on GCP specifically | [GCP deployment with Terraform](/self-hosting/gcp-terraform) | **Then configure and operate:** | If you want to... | Read | | - | - | | Confirm your machine can run Observal | [Requirements](/self-hosting/requirements) | | Get the stack running locally for dev | [Docker Compose setup](/self-hosting/docker-compose) | | Know every env var that matters | [Configuration](/self-hosting/configuration) | | See every port and volume at a glance | [Ports and volumes](/self-hosting/ports-and-volumes) | | Understand the DBs and retention | [Databases](/self-hosting/databases) | | Set up SSO, JWT keys, demo accounts | [Authentication and SSO](/self-hosting/authentication) | | Understand session delivery | [Session tracking and reconciliation](/core-concepts/session-tracking) | | Upgrade safely | [Upgrades](/self-hosting/upgrades) | | Back up and restore | [Backup and restore](/self-hosting/backup-and-restore) | | Fix something that's broken | [Troubleshooting](/self-hosting/troubleshooting) | ## Production checklist Before putting Observal in front of real users: 1. **Protect `SECRET_KEY`**: server-package setup generates it in `secrets/`; source deployments must set a strong direct value or `SECRET_KEY_FILE`. 2. **Protect database passwords**: server-package setup generates restricted PostgreSQL and ClickHouse credential files; source deployments must replace the `.env.example` defaults. 3. **Scope `CORS_ALLOWED_ORIGINS`** to your real frontend host. 4. **Configure SSO** in **Admin → SSO**, including `deployment.sso_only` if you want SSO-only login. 5. **Tune rate limits** (`RATE_LIMIT_AUTH`, `RATE_LIMIT_AUTH_STRICT`). 6. **Set `DATA_RETENTION_DAYS`** to match your retention policy (default 90 days). 7. **Back up the JWT key volume** (`apidata`): losing it invalidates every session. 8. **Remove demo accounts**: remove `DEMO_*` variables and generated demo password files before a real deployment, then delete any already-seeded demo users. Each of these is covered in the linked deep-dive below. Start with [Requirements](/self-hosting/requirements). # Ports and volumes Source: https://docs.observal.io/self-hosting/ports-and-volumes Every exposed port and persistent volume, at a glance. ## Host ports (default) | Service | Host port | Env var (to remap) | Protocol | | - | - | - | - | | API | 8000 | `API_HOST_PORT` | HTTP / WebSocket | | Web UI | 3000 | `WEB_HOST_PORT` | HTTP | | Postgres | 5432 | `POSTGRES_HOST_PORT` | TCP | | ClickHouse | 8123 | `CLICKHOUSE_HOST_PORT` | HTTP | | Redis | 6379 | `REDIS_HOST_PORT` | TCP | | Prometheus, optional | 9090 | `PROMETHEUS_HOST_PORT` | HTTP | | Grafana, optional | 3001 | `GRAFANA_HOST_PORT` | HTTP | The worker has no exposed port; it talks to Redis and ClickHouse internally only. ### Remap on startup ```bash theme={null} POSTGRES_HOST_PORT=5433 REDIS_HOST_PORT=6380 \ docker compose -f docker/docker-compose.yml up --build -d ``` ### Checking port conflicts ```bash theme={null} # macOS / Linux lsof -nP -iTCP:5432 -sTCP:LISTEN lsof -nP -iTCP:6379 -sTCP:LISTEN # Windows (PowerShell) netstat -ano | findstr :5432 ``` ## Persistent volumes All volumes are named and managed by Docker. They survive `docker compose down` but are deleted by `docker compose down -v`. | Volume | Mount point | Contents | Loss impact | | - | - | - | - | | `pgdata` | `/var/lib/postgresql/data` | Postgres data (users, registry, RBAC) | Catastrophic - all accounts, agents, MCPs lost | | `chdata` | `/var/lib/clickhouse` | ClickHouse session, audit, and security data | High: all telemetry lost; accounts and registry survive | | `redisdata` | `/data` | Redis persistence | Low - job queue lost; pending jobs need to be re-kicked | | `grafanadata`, optional | `/var/lib/grafana` | Grafana dashboards and config | Medium - custom dashboards lost; defaults are re-provisioned | | `apidata` | `/data` (API container) | **JWT signing keys** | Catastrophic - every session invalidated, users must re-login; backup/restore required | **Back up `apidata` and `pgdata` before any upgrade.** See [Backup and restore](/self-hosting/backup-and-restore). ## Security posture defaults The stack uses hardened defaults out of the box: * **Read-only root filesystem** on `observal-api`, `observal-worker`, `observal-web` (tmpfs for `/tmp`). * **`no-new-privileges`** on every service. * **Memory limits** on all services (see [Requirements](/self-hosting/requirements#hardware)). * **Health checks** on Postgres, ClickHouse, Redis, API. * **Bridge network** (`observal-net`) isolates inter-service traffic. ## Internal DNS Inside the compose network, services resolve each other by service name: | From | To | URL | | - | - | - | | API / worker | Postgres | `postgresql+asyncpg://postgres:postgres@observal-db:5432/observal` | | API / worker | ClickHouse | `clickhouse://default:clickhouse@observal-clickhouse:8123/observal` | | API / worker | Redis | `redis://observal-redis:6379` | | Web | API | `http://observal-api:8000` | The CLI running on your host machine resolves via `localhost:`. ## Next → [Databases](/self-hosting/databases) # Production deployment Source: https://docs.observal.io/self-hosting/production-deploy Deploy Observal as a distributed, highly available stack on AWS or GCP using Terraform. Best for SLA-bound deployments, teams over 50 users, and any environment where uptime and horizontal scaling matter. **End state:** a fully managed Observal install with autoscaling compute, managed databases with automated failover, encrypted secrets, centralized logging, automated backups, and HTTPS on your domain — provisioned with a single `terraform apply`. ## When to use this vs. single-node | | [Single-node](/self-hosting/single-node-deploy) | Production (this guide) | | - | - | - | | **Best for** | Small/mid teams, internal use, POCs | Enterprise, SLA-bound, high-traffic | | **Infra** | 1 VM, any cloud or on-prem | \~100 managed cloud resources | | **Cost** | \$20–150/mo | \~\$255/mo (AWS) | | **HA** | No — single point of failure | Yes — Multi-AZ databases, autoscaling compute | | **Failover** | Manual (reboot / redeploy) | Automatic (managed services handle it) | | **Scaling** | Vertical (bigger VM) | Horizontal (add more containers) | | **Backups** | Cron + S3 | Automated (RDS snapshots, systemd timers, S3 lifecycle) | | **Secrets** | Server-package files, or direct `.env` values for source Compose | SSM Parameter Store / Secret Manager (encrypted, auditable) | | **Logging** | `docker compose logs` | CloudWatch / Cloud Logging (centralized, retained, searchable) | | **Time to deploy** | 10 minutes | 20–30 minutes | ## Choose your cloud Observal ships Terraform modules for both AWS and GCP. They provision equivalent architectures using each cloud's native managed services. ### AWS ```mermaid theme={null} flowchart TB internet[Internet] alb[ALB HTTPS] api["ECS Fargate: api - 2 to 10 tasks"] web["ECS Fargate: web - 2 to 6 tasks"] worker["ECS Fargate: worker - 1 to 5 tasks"] data[EC2 data host] pg["RDS Postgres - Multi-AZ"] redis["ElastiCache Redis - 2-node"] ch["ClickHouse - EBS gp3"] grafana[Grafana] prometheus[Prometheus] internet --> alb alb -->|/api/*| api alb -->|default| web alb -->|/grafana/*| data api --> worker api --> pg api --> redis api --> ch worker --> pg worker --> redis worker --> ch data --> ch data --> grafana data --> prometheus ``` | Component | AWS Service | Notes | | - | - | - | | Compute (stateless) | ECS Fargate | api, web, worker as separate services | | Postgres | RDS Postgres 16 | Multi-AZ on prod, encrypted, automated backups | | Redis | ElastiCache Redis 7 | 2-node replication, automatic failover on prod | | ClickHouse | EC2 + EBS gp3 | Self-hosted | | Load balancer | ALB | HTTPS via ACM, path-based routing | | Secrets | SSM Parameter Store | Encrypted with KMS, injected into ECS tasks | | Logging | CloudWatch | Per-service log groups | | Backups | S3 + RDS snapshots | Lifecycle: Standard → IA → Glacier → expire | | DNS | Route 53 | Optional; works without a custom domain | **Full walkthrough:** [AWS deployment with Terraform](/self-hosting/aws-terraform) **Quick start:** ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal/infra/terraform/aws # Configure cp terraform.tfvars.example terraform.tfvars # Edit: region, environment, domain_name (optional), alb_ingress_cidrs # Deploy terraform init terraform plan -out tf.plan terraform apply tf.plan # Verify terraform output app_url curl -fsS "$(terraform output -raw app_url)/readyz" ``` Apply takes ~~20–30 minutes (RDS Multi-AZ provisioning dominates). Baseline cost: \*\*~~\$255/mo\*\* in us-east-1. ### GCP ```mermaid theme={null} flowchart TB internet[Internet] lb["Global HTTPS LB - managed SSL"] api[Cloud Run: api] web[Cloud Run: web] init["Cloud Run Job: init - migrations"] worker[Cloud Run: worker] data["GCE data host - IAP"] pg["Cloud SQL - Postgres"] redis["Memorystore - Redis"] ch["ClickHouse - Persistent Disk"] grafana[Grafana] prometheus[Prometheus] internet --> lb lb -->|/api/*| api lb -->|default| web lb -->|/grafana/*| data init --> pg api --> worker api --> pg api --> redis api --> ch worker --> pg worker --> redis worker --> ch data --> ch data --> grafana data --> prometheus ``` | Component | GCP Service | Notes | | - | - | - | | Compute (stateless) | Cloud Run v2 | api, web, worker as separate services | | Migrations | Cloud Run Jobs | One-shot init task | | Postgres | Cloud SQL | HA configuration available | | Redis | Memorystore | Managed Redis | | ClickHouse | GCE instance | Docker Compose on a single VM | | Load balancer | Global HTTPS LB | Managed SSL certificate | | Secrets | Secret Manager | Injected into Cloud Run at start | | Logging | Cloud Logging | Built-in, no config needed | | Backups | GCS | Versioned bucket with lifecycle | | DNS | Cloud DNS | Optional | | Shell access | IAP SSH tunnel | No public SSH, no SSH keys | **Full walkthrough:** [GCP deployment with Terraform](/self-hosting/gcp-terraform) **Quick start:** ```bash theme={null} cd Observal/infra/terraform/gcp # Enable required APIs gcloud services enable \ run.googleapis.com sqladmin.googleapis.com redis.googleapis.com \ compute.googleapis.com secretmanager.googleapis.com \ dns.googleapis.com vpcaccess.googleapis.com \ servicenetworking.googleapis.com # Configure cp terraform.tfvars.example terraform.tfvars # Edit: project_id, region, domain_name (optional) # Deploy terraform init terraform plan -out tf.plan terraform apply tf.plan # Run migrations gcloud run jobs execute observal-prod-init --region=us-central1 # Verify terraform output app_url ``` ## Day-2 operations These apply to both AWS and GCP. Cloud-specific commands are in the respective guides. ### Upgrade to a new release ```hcl theme={null} # terraform.tfvars image_tag = "v1.5.0" ``` ```bash theme={null} terraform apply ``` The init task re-runs migrations. Compute services roll over with zero downtime. ### Roll back Set `image_tag` to the previous version and `terraform apply`. Database data is not affected. If the failed version ran a destructive migration (rare, always called out in the CHANGELOG), restore from the pre-upgrade database backup. ### Scale up Adjust variables in `terraform.tfvars`: ```hcl theme={null} # More API containers api_desired_count = 4 api_autoscale_max = 20 # Bigger database db_instance_class = "db.m6g.large" # AWS # postgres_tier = "db-custom-4-16384" # GCP # Bigger ClickHouse host data_instance_type = "m6i.xlarge" # AWS # data_machine_type = "e2-standard-4" # GCP ``` ```bash theme={null} terraform apply ``` ### Tear down ```bash theme={null} terraform destroy ``` On prod, databases have deletion protection enabled. Disable manually before destroy if you really mean it. ## Cost comparison ### AWS (us-east-1, on-demand) | Component | Spec | \~\$/month | | - | - | - | | Fargate api 2× | 0.5 vCPU / 1 GB | \$30 | | Fargate web 2× | 0.25 vCPU / 0.5 GB | \$15 | | Fargate worker 1× | 0.5 vCPU / 1 GB | \$15 | | EC2 data host | t3.large | \$60 | | RDS Postgres Multi-AZ | db.t4g.small | \$50 | | ElastiCache Redis 2× | cache.t4g.micro | \$25 | | ALB | — | \$20 | | NAT Gateway | — | \$33 + egress | | EBS gp3 100 GB | — | \$8 | | S3 backups | \~1 GB cold | \$0.10 | | **Total** | | **\~\$255** | Set `environment = "staging"` to halve the bill (single-AZ RDS, one Redis node, no deletion protection). ### GCP (us-central1, on-demand) | Component | Spec | \~\$/month | | - | - | - | | Cloud Run api | 1 vCPU / 512 MB, min 1 instance | \$25 | | Cloud Run web | 1 vCPU / 256 MB, min 1 instance | \$15 | | Cloud Run worker | 1 vCPU / 512 MB, min 1 instance | \$25 | | Cloud SQL Postgres | db-f1-micro (shared) | \$10 | | Memorystore Redis | 1 GB basic | \$35 | | GCE data host | e2-standard-2 | \$50 | | Global HTTPS LB | — | \$18 | | Persistent Disk 100 GB | — | \$4 | | GCS backups | \~1 GB cold | \$0.02 | | **Total** | | **\~\$180** | GCP is typically cheaper due to Cloud Run's per-request billing and no NAT Gateway equivalent charge. ## Production hardening checklist Applies to both clouds. See the cloud-specific guides for implementation details. * [ ] Enable remote Terraform state (S3 + DynamoDB on AWS, GCS on GCP) * [ ] Restrict load balancer ingress to known CIDRs * [ ] Enable cloud security services (GuardDuty + Config on AWS, Security Command Center on GCP) * [ ] Set up alerting on database CPU, memory, and 5xx error rates * [ ] Attach a WAF to the load balancer * [ ] Enable Redis transit encryption * [ ] Configure [SSO](/self-hosting/authentication) (SAML or OIDC) * [ ] Test the [backup and restore](/self-hosting/backup-and-restore) procedure end-to-end * [ ] Replace the GitHub tarball pull in the data host bootstrap with an artifact you control ## Next * [AWS deployment with Terraform](/self-hosting/aws-terraform) — full AWS walkthrough * [GCP deployment with Terraform](/self-hosting/gcp-terraform) — full GCP walkthrough * [Configuration](/self-hosting/configuration) — environment variables reference * [Upgrades](/self-hosting/upgrades) — upgrade and rollback procedures * [Backup and restore](/self-hosting/backup-and-restore) — detailed restore procedures # Requirements Source: https://docs.observal.io/self-hosting/requirements Minimum and recommended specs for running the Observal stack. ## Hardware | Profile | CPU | Memory | Disk | Who it's for | | - | - | - | - | - | | Developer | 2 vCPU | 4 GB | 20 GB | One engineer, local machine | | Small team (≤10) | 2 vCPU | 6 GB | 50 GB | Small deployment, moderate telemetry | | Team (10–50) | 4 vCPU | 12 GB | 200 GB + fast SSD | Typical production deployment | | Large team (50+) | 8+ vCPU | 32 GB | 500 GB + fast SSD | High telemetry volume; consider externalizing ClickHouse | Default Docker memory limits (the API, worker and web limits differ between source Compose and the one-line server package): | Service | Source Compose | Server package | | - | - | - | | `observal-api` | 1536 MB | 512 MB | | `observal-worker` | 1536 MB | 512 MB | | `observal-web` | 32 MB | 256 MB | | `observal-db` | 512 MB | 512 MB | | `observal-clickhouse` | 2 GB | 2 GB | | `observal-redis` | 256 MB | 256 MB | | `observal-lb` | 64 MB | 64 MB | | `observal-grafana` (optional overlay) | 512 MB | 512 MB | ClickHouse has the largest memory limit. Tune `CLICKHOUSE_MEMORY_LIMIT` for your workload; see [resource tuning](resource-tuning.md). ## Disk: where the data goes The heaviest user of disk is **ClickHouse**. Growth depends on: * Number and length of harness sessions * Raw transcript record size * `DATA_RETENTION_DAYS` Session transcripts vary significantly by harness and tool output size. Measure representative workloads and plan 2 to 3 times headroom over observed growth. Postgres stays under 500 MB for most deployments; it holds only registry metadata and user accounts. ## Software | Software | Version | Notes | | - | - | - | | Docker | ≥ 24.0 | With Compose v2 (`docker compose`, not `docker-compose`) | | Linux / macOS host | any modern | Windows via WSL2 works | | Bash / zsh | any | For the CLI install | > [!NOTE] > > Homebrew's Docker formula is outdated and may ship an older Compose version. Install [Docker Desktop](https://docs.docker.com/get-docker/) or use your distro's upstream packages to get Docker Engine ≥ 24.0 with Compose v2. For the **CLI** (developer machines, not the server): * **Standalone binary** (recommended) -- no dependencies, just `curl | bash` * Or Python **3.11, 3.12, or 3.13** with `uv`, `pipx`, or `pip` ## Network * **Outbound HTTPS**: required for image downloads and enabled integrations such as OAuth, webhooks, external Git repositories, or model providers. * **Inbound**: the server package routes the UI, API, and session telemetry through nginx on port `8000`. * **Between services**: the private `observal-net` bridge handles application traffic. * **Host bindings**: new server-package installs bind nginx, the direct web port, PostgreSQL, ClickHouse, Redis, Prometheus, and Grafana to `127.0.0.1` by default. * **Secret access**: server-package setup records the operator's group as `OBSERVAL_SECRET_GID`; only that group is added to containers that need their service-specific files. ## TLS / HTTPS Local loopback access can use HTTP. For every remote deployment, terminate TLS 1.2 or later with Caddy, nginx, an enterprise load balancer, or a cloud ingress. Point the proxy at the loopback nginx listener on `localhost:8000`. Example Caddy configuration: ```caddyfile theme={null} observal.your-company.internal { reverse_proxy localhost:8000 } ``` Set `OBSERVAL_BIND_ADDRESS` to a non-loopback address only when an external TLS proxy cannot reach loopback, or when plaintext access is an intentional private-network risk. Existing server-package installations retain their previous bind address during setup upgrades. See the [security assurance case](/security/assurance-case) for trust boundaries and residual network risks. ## Next → [Docker Compose setup](/self-hosting/docker-compose) # Single-node deployment Source: https://docs.observal.io/self-hosting/single-node-deploy Run the full Observal stack on a single VM with Docker Compose. Best for teams of up to \~50 users, internal tools, evaluations, and any deployment where simplicity matters more than multi-AZ redundancy. **End state:** Observal running behind TLS on one server, with automated backups to S3-compatible storage, surviving reboots, and upgradable in under a minute. ## When to use this vs. the Terraform module | | Single-node (this guide) | [Production Terraform](/self-hosting/aws-terraform) | | - | - | - | | **Best for** | Small/mid teams, internal use, POCs | Enterprise, SLA-bound, high-traffic | | **Infra** | 1 VM, any cloud or on-prem | \~100 managed AWS resources | | **Cost** | \$20–150/mo | \~\$255/mo | | **HA** | No — single point of failure | Yes — Multi-AZ Postgres, autoscaling ECS | | **Time to deploy** | 10 minutes | 20–30 minutes | | **Operational complexity** | Low — SSH, docker compose, cron | Medium — Terraform, AWS console, CloudWatch | | **Scaling** | Vertical (bigger VM) | Horizontal (more Fargate tasks) | ## Architecture ```mermaid theme={null} flowchart TB subgraph vm[Single VM] nginx["nginx - TLS"] web[Web UI] api["API - FastAPI"] worker["Worker - arq"] pg[(Postgres)] redis[(Redis)] ch[(ClickHouse)] grafana[Grafana] systemd["systemd - docker compose restart"] cron["cron - daily backups to S3"] end nginx --> web nginx --> api api --> pg api --> ch api --> redis api --> worker worker --> pg worker --> redis worker --> ch grafana --> ch systemd -.-> nginx systemd -.-> api systemd -.-> worker cron -.-> pg cron -.-> ch ``` Everything runs as Docker containers on a single host. The nginx LB routes traffic and terminates TLS. Docker's restart policy and systemd keep the stack running across reboots. ## Prerequisites | Requirement | Minimum | Recommended | | - | - | - | | **VM** | 2 vCPU, 4 GB RAM, 40 GB SSD | 4 vCPU, 8 GB RAM, 100 GB SSD | | **OS** | Ubuntu 22.04+ / Amazon Linux 2023 / Debian 12 | Ubuntu 24.04 LTS | | **Docker** | Engine ≥ 24.0 with Compose v2 | Latest stable | | **Domain** (for TLS) | A DNS record pointing to the VM's public IP | — | | **Firewall** | Ports 80, 443 open inbound | — | ### Sizing guide | Team size | VM spec | Estimated cost (AWS) | | - | - | - | | 1–10 users | `t3.medium` (2 vCPU / 4 GB) | \~\$30/mo | | 10–30 users | `t3.large` (2 vCPU / 8 GB) | \~\$60/mo | | 30–50 users | `t3.xlarge` (4 vCPU / 16 GB) | \~\$120/mo | | 50+ users | Consider the [Terraform module](/self-hosting/aws-terraform) | \~\$255/mo | ClickHouse is the memory consumer. If you run out, increase `CLICKHOUSE_MEMORY_LIMIT` before resizing the VM. ## Step 1: Provision the VM Use any cloud provider or on-prem hypervisor. Example for AWS: ```bash theme={null} aws ec2 run-instances \ --image-id ami-0c7217cdde317cfec \ --instance-type t3.large \ --key-name your-key \ --security-group-ids sg-xxxxx \ --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":100,"VolumeType":"gp3"}}]' \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=observal}]' ``` For other clouds: * **GCP:** `e2-standard-2` with 100 GB balanced persistent disk * **Azure:** `Standard_B2ms` with 100 GB Premium SSD * **Hetzner:** `CPX31` (4 vCPU / 8 GB, \~€15/mo) * **On-prem:** any Linux box meeting the specs above ## Step 2: Install Docker SSH into the VM and install Docker: ```bash theme={null} ssh ubuntu@your-server-ip # Ubuntu / Debian curl -fsSL https://get.docker.com | sh sudo usermod -aG docker $USER newgrp docker # Verify docker version docker compose version ``` ## Step 3: Clone and configure ```bash theme={null} git clone https://github.com/Observal/Observal.git cd Observal cp .env.example .env ``` Edit `.env` for production. At minimum, change these: ```bash theme={null} # Generate a secure secret key SECRET_KEY=$(python3 -c "import secrets; print(secrets.token_urlsafe(32))") # Set strong database passwords POSTGRES_PASSWORD=$(openssl rand -base64 24) CLICKHOUSE_PASSWORD=$(openssl rand -base64 24) # Set your domain (used for CORS and OAuth redirects) CORS_ALLOWED_ORIGINS=https://observal.yourcompany.com FRONTEND_URL=https://observal.yourcompany.com # Remove demo accounts for production # Comment out or delete all DEMO_* variables ``` Write them into `.env`: ```bash theme={null} sed -i "s|^SECRET_KEY=.*|SECRET_KEY=$SECRET_KEY|" .env sed -i "s|^POSTGRES_PASSWORD=.*|POSTGRES_PASSWORD=$POSTGRES_PASSWORD|" .env sed -i "s|^CLICKHOUSE_PASSWORD=.*|CLICKHOUSE_PASSWORD=$CLICKHOUSE_PASSWORD|" .env ``` > SAML SSO, audit logs, and executive dashboards are included in the open-source distribution. See [Configuration](/self-hosting/configuration). For mounted credentials and private keys, use the documented [`NAME_FILE` inputs](/self-hosting/configuration#secret-files) instead of placing secret contents in `.env`. ## Step 4: Set up TLS This guide uses the source Compose stack. Every `localhost:80` example below refers to that stack's nginx load balancer. Do not expose its HTTP listener directly to remote clients. For downloadable server-package bindings, see [Requirements](/self-hosting/requirements#tls--https). ### Option A: Caddy (simplest — automatic HTTPS) ```bash theme={null} sudo apt install -y caddy ``` Create `/etc/caddy/Caddyfile`: ```caddyfile theme={null} observal.yourcompany.com { reverse_proxy localhost:80 } ``` ```bash theme={null} sudo systemctl enable --now caddy ``` Caddy handles Let's Encrypt certificates automatically. No renewal cron needed. ### Option B: Certbot + nginx production config ```bash theme={null} sudo apt install -y certbot sudo certbot certonly --standalone -d observal.yourcompany.com ``` Then use the production compose overlay which configures nginx for TLS: ```bash theme={null} cd docker docker compose -f docker-compose.yml -f docker-compose.production.yml up -d ``` Set up auto-renewal: ```bash theme={null} echo "0 3 * * * certbot renew --quiet && docker compose -f /home/ubuntu/Observal/docker/docker-compose.yml -f /home/ubuntu/Observal/docker/docker-compose.production.yml restart observal-lb" | sudo tee -a /etc/crontab ``` ### Option C: Behind a cloud load balancer If your VM sits behind an AWS ALB, GCP HTTPS LB, or Cloudflare, terminate TLS there and proxy to port 80 on the VM. No TLS config on the VM itself. ## Step 5: Start the stack ```bash theme={null} cd ~/Observal docker compose -f docker/docker-compose.yml up -d --build ``` First build takes 3–5 minutes (pulling images, building the API and web containers). Watch the logs: ```bash theme={null} docker compose -f docker/docker-compose.yml logs -f observal-init observal-api ``` Wait for: ``` observal-init | INFO - Database up to date. observal-api | INFO - Application startup complete. ``` Verify: ```bash theme={null} curl -fsSL http://localhost/health # {"status":"ok","initialized":true} ``` ## Step 6: Survive reboots Docker's `restart: unless-stopped` policy (already set in `docker-compose.yml`) handles container restarts. To ensure Docker itself starts on boot: ```bash theme={null} sudo systemctl enable docker ``` Test it: ```bash theme={null} sudo reboot # After reconnecting: docker compose -f ~/Observal/docker/docker-compose.yml ps # All services should be running ``` ## Step 7: Set up automated backups Create a backup script: ```bash theme={null} sudo mkdir -p /opt/observal-backups cat << 'SCRIPT' | sudo tee /opt/observal-backups/backup.sh #!/bin/bash set -euo pipefail BACKUP_DIR="/opt/observal-backups" DATE=$(date +%Y%m%d-%H%M) COMPOSE="docker compose -f /home/ubuntu/Observal/docker/docker-compose.yml" # Postgres $COMPOSE exec -T observal-db pg_dump -U postgres observal | gzip > "$BACKUP_DIR/pg-$DATE.sql.gz" # JWT keys $COMPOSE exec -T observal-api tar czf - -C /data keys > "$BACKUP_DIR/keys-$DATE.tar.gz" # Prune backups older than 30 days find "$BACKUP_DIR" -name "pg-*.sql.gz" -mtime +30 -delete find "$BACKUP_DIR" -name "keys-*.tar.gz" -mtime +30 -delete # Upload to S3 (optional — install awscli first) # aws s3 sync "$BACKUP_DIR" s3://your-backup-bucket/observal/ --exclude "backup.sh" echo "Backup completed: $DATE" SCRIPT sudo chmod +x /opt/observal-backups/backup.sh ``` Schedule it: ```bash theme={null} echo "0 3 * * * root /opt/observal-backups/backup.sh >> /var/log/observal-backup.log 2>&1" | sudo tee /etc/cron.d/observal-backup ``` For ClickHouse (weekly, since it's larger): ```bash theme={null} echo "0 4 * * 0 root docker compose -f /home/ubuntu/Observal/docker/docker-compose.yml exec -T observal-clickhouse clickhouse-client --password \$CLICKHOUSE_PASSWORD --query \"BACKUP DATABASE observal TO Disk('backups', 'weekly-\$(date +\%Y\%m\%d).zip')\" >> /var/log/observal-backup.log 2>&1" | sudo tee /etc/cron.d/observal-ch-backup ``` See [Backup and restore](/self-hosting/backup-and-restore) for detailed restore procedures. ## Step 8: First login Install the CLI on your local machine: ```bash theme={null} curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash ``` Log in: ```bash theme={null} observal auth login # Server URL: https://observal.yourcompany.com # Email: (create your admin account or use demo creds if you kept them) ``` Verify: ```bash theme={null} observal auth whoami observal auth status ``` ## Upgrades ```bash theme={null} cd ~/Observal git fetch --tags git checkout v1.5.0 # or whatever version docker compose -f docker/docker-compose.yml up -d --build ``` The Compose init service applies migrations before API startup. See [Upgrades](/self-hosting/upgrades) for rollback procedures. Or use the CLI: ```bash theme={null} observal server upgrade --version 1.5.0 --force --output json ``` ## Monitoring Prometheus and Grafana are optional. Start the core stack with Prometheus only: ```bash theme={null} docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up -d ``` Start Prometheus and Grafana: ```bash theme={null} COMPOSE_PROFILES=grafana docker compose -f docker/docker-compose.yml -f docker/docker-compose.observability.yml up -d ``` Access Grafana at `http://your-server:3001` when the Grafana profile is enabled. For basic alerting without Grafana, add a health check cron: ```bash theme={null} echo "*/5 * * * * root curl -fsS http://localhost/health > /dev/null || echo 'Observal health check failed' | mail -s 'ALERT: Observal down' ops@yourcompany.com" | sudo tee /etc/cron.d/observal-health ``` ## Security hardening The [security assurance case](/security/assurance-case) describes the deployment trust boundaries and residual risks. Before exposing to the internet: * [ ] Remove all `DEMO_*` env vars from `.env` * [ ] Set a strong `SECRET_KEY` * [ ] Restrict SSH access (key-only, no password auth) * [ ] Configure a firewall (`ufw allow 80,443/tcp && ufw enable`) * [ ] Set up [SSO](/self-hosting/authentication) if available * [ ] Enable automatic security updates (`sudo apt install unattended-upgrades`) * [ ] Bind database ports to localhost only (already done in `docker-compose.yml` via `127.0.0.1:` prefix) ## Scaling up When you outgrow a single node: | Symptom | Fix | | - | - | | API response times increasing | Increase `API_WORKERS` in `.env` (default 2), or bump to a bigger VM | | ClickHouse queries slow | Increase `CLICKHOUSE_MEMORY_LIMIT` or move to a bigger VM | | Disk filling up | Reduce `DATA_RETENTION_DAYS`, add a bigger disk, or move ClickHouse data to a separate volume | | Need HA / zero downtime deploys | Migrate to the [Terraform module](/self-hosting/aws-terraform) | ## Next * [Configuration](/self-hosting/configuration) — all environment variables * [Backup and restore](/self-hosting/backup-and-restore) — detailed restore procedures * [Upgrades](/self-hosting/upgrades) — safe upgrade and rollback flow * [Troubleshooting](/self-hosting/troubleshooting) — common issues # Troubleshooting Source: https://docs.observal.io/self-hosting/troubleshooting Common failure modes and their fixes. If none of these match, open a [GitHub Discussion](https://github.com/Observal/Observal/discussions) with the output of `observal auth status` and relevant logs from `docker compose logs`. ## Install and CLI ### `"Connection failed. Is the server running?"` The CLI cannot reach the API. Check: ```bash theme={null} docker compose -f docker/docker-compose.yml ps # API status curl http://localhost/health # API health observal config show # is server_url right? ``` If `server_url` is wrong: ```bash theme={null} observal config set server_url http://localhost observal auth login ``` ### `"System already initialized"` when logging in The server already has users, so bootstrap is disabled. Use `observal auth login` with an email + password or an API key, not a fresh bootstrap flow. ## Docker and networking ### `port is already allocated` Another process is on one of Observal's default ports. Remap host ports: ```bash theme={null} POSTGRES_HOST_PORT=5433 REDIS_HOST_PORT=6380 \ docker compose -f docker/docker-compose.yml up --build -d ``` Full list in [Ports and volumes](/self-hosting/ports-and-volumes). ### Service stuck in `starting` The API depends on Postgres, ClickHouse, and Redis being healthy. Check each: ```bash theme={null} docker compose -f docker/docker-compose.yml ps docker compose -f docker/docker-compose.yml logs observal-db docker compose -f docker/docker-compose.yml logs observal-clickhouse docker compose -f docker/docker-compose.yml logs observal-redis ``` Common causes: * ClickHouse stuck during initial `CREATE TABLE`. Restart it once the healthcheck passes on other DBs * `CLICKHOUSE_PASSWORD` mismatch between services and API config ### Services restart in a loop Check logs (`docker compose logs -f `). Three frequent causes: * Memory limit too tight. Bump limits in `docker-compose.yml` * Corrupt volume. Wipe and restore from backup * Config error introduced during an upgrade. Roll back ## Auth ### Admin forgot password There is no reset code to look up. An admin resets the account directly: ```bash theme={null} observal admin reset-password admin@demo.example --generate ``` Hand the printed password to the user out-of-band; they must change it at their next sign-in. ### OAuth login fails with `redirect_uri_mismatch` The IdP doesn't have the right redirect URI registered. Add: ``` {FRONTEND_URL}/api/v1/auth/oauth/callback ``` with `FRONTEND_URL` set to your real external URL (scheme and host must match exactly). ### All users logged out after restart Likely the `apidata` volume was recreated, so the JWT signing keys are new. Restore the `apidata` volume from backup, or accept that all sessions are invalid and everyone has to log in again. ## Telemetry ### Nothing in the dashboard Run through, in order: ```bash theme={null} # 1. Are sessions arriving at all? observal ops telemetry status # 2. Are session hooks installed for the harness? observal doctor --output json # 3. Is the API reachable from the harness environment? curl http://localhost/health ``` If hooks are missing, run `observal doctor patch --harness `. If sessions still are not arriving, check `~/.observal/telemetry_buffer.db`; growth indicates pending session delivery rather than silent loss. ### ClickHouse not receiving data Check the `CLICKHOUSE_URL` the API is using: ```bash theme={null} docker compose -f docker/docker-compose.yml exec observal-api \ printenv CLICKHOUSE_URL ``` The source Compose default is `clickhouse://default:clickhouse@observal-clickhouse:8123/observal`. Mismatches typically happen after changing `CLICKHOUSE_PASSWORD` without updating the URL. Server-package installs use `CLICKHOUSE_URL_FILE=/run/secrets/clickhouse_url`, a hashed ClickHouse user configuration, and a separate health-check password file. Confirm the file is mounted without printing it: ```bash theme={null} docker compose exec observal-api test -r /run/secrets/clickhouse_url docker compose exec observal-clickhouse test -r /run/secrets/clickhouse_password ``` Verify ClickHouse itself: ```bash theme={null} docker compose -f docker/docker-compose.yml exec observal-clickhouse \ clickhouse-client --query "SELECT count() FROM observal.session_events" ``` ## Web UI ### Blank white page Frontend is still building. Check: ```bash theme={null} docker compose -f docker/docker-compose.yml logs -f observal-web ``` For local dev (running Vite outside Docker), the dev server proxies `/api` to `http://localhost:8000` (see `web/vite.config.ts`); confirm your backend is reachable there. ### Login redirects back to login immediately Browser cookies aren't being set. Usually one of: * `FRONTEND_URL` doesn't match the URL you're hitting. * `CORS_ALLOWED_ORIGINS` doesn't include your frontend origin. * You're on HTTP behind a proxy that's setting `secure` cookies. Terminate TLS at your proxy and keep `FRONTEND_URL=https://...`. ## Where to get more help * Logs: `docker compose -f docker/docker-compose.yml logs -f` * Health: `curl http://localhost/health` * Status: `observal auth status` * Community: [GitHub Discussions](https://github.com/Observal/Observal/discussions) * Bugs: [GitHub Issues](https://github.com/Observal/Observal/issues). Please use Discussions for questions, Issues only for confirmed bugs # Upgrades Source: https://docs.observal.io/self-hosting/upgrades Safe upgrade flow for the Observal server stack. > [!IMPORTANT] > > When upgrading into the release that introduces complete-folder skill delivery, follow the [skill-folder rollout procedure](/skill-folder-rollout) instead of using the quick, standard, or blue/green instructions below unchanged. Those paths do not guarantee that old API and worker processes have stopped before the new init container runs PostgreSQL migrations. Keep folder delivery off, back up and rehearse the historical upgrade, and schedule a maintenance window to drain old processes **before** migration; enable delivery only after a homogeneous rollout and separate approval. ## Quick upgrade (recommended) If you installed Observal via `observal server start` (the embedded stack), the CLI handles upgrades automatically: ```bash theme={null} observal server upgrade --dry-run --output json observal server upgrade --version 0.9.0 --force --output json ``` This pulls new Docker images, backs up PostgreSQL, recreates containers, and runs health checks. If the health check fails, it requests the previous image version again. Local shell and Docker access authorize the operation; the command does not require a reachable API or API role. See [`observal server upgrade`](/cli/server#observal-server-upgrade) for full details. ### Server-package upgrades If you installed with `install-server.sh`, back up `.env`, `secrets/`, and the `apidata` and `pgdata` volumes before rerunning the installer for the target release. Setup detects the existing `.env`. Choose the default **No** response to preserve custom values and existing direct credentials. **This only stages the new package and retains a backup: no containers restart and no migrations run.** An exit code of zero means staging succeeded, not that the server has been upgraded. The installer records the prior bind address when an older install has no `OBSERVAL_BIND_ADDRESS` setting. For the complete-folder release, follow the [supervised rollout](/skill-folder-rollout): keep the delivery gate off, stop *all* old API and worker processes before starting the new init container, then run Compose from the installed package with the same optional observability overlay/profile used by the old deployment. For a core-only package installation, after staging the package and verifying backups and the gate: ```bash theme={null} cd /opt/observal # or the directory passed to --install-dir # Set OBSERVAL_VERSION in the preserved .env to the selected release's image tag. # For installer --version vX.Y.Z, the image tag is X.Y.Z (no leading v). # An old pinned tag keeps serving old images; latest also requires an explicit pull. docker compose -f docker-compose.yml --env-file .env stop observal-api observal-worker docker compose -f docker-compose.yml --env-file .env pull observal-init observal-api observal-worker observal-web docker compose -f docker-compose.yml --env-file .env up -d --wait --wait-timeout 300 docker compose -f docker-compose.yml --env-file .env restart observal-lb curl -fsS http://127.0.0.1:8000/health # adjust for the configured API_HOST_PORT ``` Add `-f docker-compose.observability.yml` and `--profile grafana` if those were used previously; do not silently drop optional services. Verify the API externally through the load balancer, not only via its container health check. Choosing **Yes** at the configuration prompt instead intentionally rebuilds the general configuration from the new template and starts services; for this release, **stop old API and worker processes before running the installer**, because setup will start the new init container. Core application, PostgreSQL, ClickHouse, Grafana, and demo credentials are preserved or migrated into restricted files, but unrelated custom environment entries must be reapplied. ## Before a manual upgrade 1. **Back up `pgdata`** and **`apidata`**. See [Backup and restore](/self-hosting/backup-and-restore). Backing up `chdata` is nice-to-have; losing telemetry is painful but not catastrophic. 2. **Read the [CHANGELOG](https://github.com/Observal/Observal/blob/main/CHANGELOG.md)** for the releases you're jumping across. Note any breaking changes. 3. **Pin the version you're upgrading to**: don't `git pull main` blindly. Check out a release tag or a known-good commit. ## Standard upgrade ```bash theme={null} cd Observal # Fetch and check out the target version git fetch --tags git checkout v0.9.1 # Rebuild images docker compose -f docker/docker-compose.yml pull docker compose -f docker/docker-compose.yml up --build -d # Nginx retains resolved upstream IPs across API/web container recreation. # Wait for the new API to be healthy, then refresh the load balancer. docker compose -f docker/docker-compose.yml exec observal-api python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" docker compose -f docker/docker-compose.yml restart observal-lb # Verify through the public load balancer docker compose -f docker/docker-compose.yml ps curl http://localhost/health ``` The init container applies pending Postgres Alembic migrations and ClickHouse SQL migrations before the API starts. Watch the init logs for migration output: ```bash theme={null} docker logs -f observal-init # Running database migrations... # Running ClickHouse migrations... ``` ## Zero-downtime upgrade (small teams) If you run a single instance and have a \~30-second maintenance window: 1. Back up `pgdata`, `apidata`, `chdata`. 2. Stop the API and worker: `docker compose stop observal-api observal-worker`. 3. Apply migrations out of band with `alembic upgrade head` and `python -m services.clickhouse.migrations` from `observal-server`, or run the init container once. 4. Pull/rebuild new images: `docker compose pull && docker compose build observal-api observal-worker`. 5. Start: `docker compose up -d`; wait for the new API to become healthy, then `docker compose restart observal-lb` so nginx resolves the new container IPs. 6. Smoke test through the public load balancer: `curl -fsS http://localhost/health`, then `observal auth status --output json && observal ops telemetry status --output json`. Web UI, Postgres, ClickHouse, Redis stay up throughout. Users see a brief API outage (\~15–30 s). ## Zero-downtime at scale For blue/green upgrades on large deployments: 1. Run a second stack (`docker-compose.yml` with different project name and host ports) behind a reverse proxy. 2. Apply migrations before traffic cutover. Alembic handles Postgres and `services.clickhouse.migrations` handles ClickHouse. Additive migrations should work with API N-1 and N during the rollout. 3. Bring up the green stack pointing at the same `pgdata` / `chdata` / `apidata` volumes. 4. Flip the reverse proxy to green. 5. Decommission blue. If a migration is **not** additive (rare, but happens: column drops, type changes), it gets called out in the CHANGELOG and requires a brief outage. Plan the window. ## Rolling back If the new version breaks: 1. `docker compose -f docker/docker-compose.yml down` 2. `git checkout ` 3. `docker compose -f docker/docker-compose.yml up --build -d` **The catch:** if the failing version already applied a migration, downgrading to the previous API version may leave you running against a schema it doesn't know about. Options: * **If the migration is additive** (most are): the previous version works fine against the newer schema. * **If the migration is destructive**: restore from the pre-upgrade `pgdata` backup. This is why the backup is step 1. ## CLI upgrades CLI upgrades are independent of server upgrades. Users: ```bash theme={null} observal self upgrade ``` The CLI speaks a stable contract with the server. A newer CLI works against an older server and vice versa, within a release or two. On the first command after upgrading from a release that generated `observal-shim` entries, the CLI restores the original MCP command in known home and current-project config files. It writes a `.pre-unshim.bak` copy before each change. A malformed wrapped entry stops the command with a visible migration error rather than guessing at the original executable. ## Zero-downtime for the web UI The web UI is a static Vite build and restarts instantly. Users see a brief reload if they are on the page during the deploy. No special handling required. ## Next → [Backup and restore](/self-hosting/backup-and-restore) # Usage reporting Source: https://docs.observal.io/self-hosting/usage-reporting Observal can send aggregate usage reports to the Observal-operated collector at `https://usage.observal.io/api/v1/usage-pings`. Reporting is enabled by default with a six-hour cadence and starts on new installations without any setup. Until a super administrator sets a company name, reports identify the installation as `unknown-` followed by a six-digit number derived from its installation ID. Once a name is set, the next report carries it under the same installation ID. The collector and reporting dashboard are maintained separately in [Observal Usage](https://github.com/Observal/observal_usage). ## Configure reporting A super administrator can open **Admin > Settings > Usage Reporting**, which appears near the top of the settings page, to enter the company name and save the reporting configuration. Setting **Deployment Public URL** adds the deployment hostname to reports; without it the hostname is reported as `unknown`. The available frequencies are every six hours, daily, and weekly. Every six hours is the default. The same panel can disable reporting, change the frequency, preview the exact payload, and send a test report. Only a super administrator can change the company identity, frequency, or consent setting. The first upgrade to 1.15.1 or later resets usage reporting to its defaults (on, every six hours) on every existing installation, including installations where it was previously switched off. A super administrator can switch it off again after the upgrade. Disabling the switch stops scheduled delivery immediately. It does not remove reports already received by Observal. Contact the Observal maintainers with the installation ID shown in the panel to request deletion. ## Data included * Stable, randomly generated installation ID * Company name and deployment hostname supplied by the administrator, or the `unknown-` placeholder and `unknown` hostname when unset * Observal version and deployment type * Aggregate counts for users, teams, registry components, agent installations, and sessions * Aggregate session totals by harness * Distinct active-user and active-agent counts for 7- and 30-day windows * Aggregate event, prompt, tool-call, tool-result, token, cache-token, and credit totals for 7- and 30-day windows * Thirty-day session averages for duration, prompts, and tool calls * Thirty-day counts for sessions using tools or tokens, registered versus unregistered agents, top-level versus sub-agent sessions, and distinct agent/model versions * Aggregate parser-error and truncated-event counts for ingestion health * Boolean adoption signals for selected server features * Report schema version and timestamp ## Data never included Usage reports do not include names, email addresses, user IDs, agent IDs, prompts, responses, tool arguments, tool results, trace events, source code, file paths, repository names, model identifiers, IP addresses, authentication tokens, API keys, credentials, or arbitrary configuration values. Token and credit fields contain numeric totals only. Distinct model and agent-version fields contain counts, never their names or identifiers. Only the selected feature flags listed above are included. ## Delivery behavior The selected schedule uses fixed UTC boundaries: * Every six hours: 00:30, 06:30, 12:30, and 18:30 UTC * Daily: 06:30 UTC each day * Weekly: Monday at 06:30 UTC The worker evaluates due reports every six hours. If a scheduled report was missed or failed, a later worker pass retries it until the current schedule window has a successful delivery. Each pass makes up to three total delivery attempts with short backoff. A failure never blocks normal Observal operation. The last successful send, latest error, selected frequency, and next delivery time are visible in the Usage Reporting settings panel. The collector URL is fixed in production releases. `USAGE_PING_URL` exists only to direct development and isolated test deployments to a local collector. # Complete-folder skill rollout (operators) Source: https://docs.observal.io/skill-folder-rollout `registry.skill_folder_delivery_enabled` is a **deployment-level rollout control**, not a permission that authors request for every skill. After a verified rollout it should remain enabled for normal operation: authors save drafts, explicitly submit exact revisions, reviewers inspect those revisions, and consumers install approved versions. The default is **off** because enabling it while older API/worker processes or clients are still running can allow reviews that some installations cannot safely deliver. Do not change the default just to make a test pass. ## Before enabling * Obtain an independently reviewed release and explicit operator authorization. The PR being green, a fresh database, and a browser demonstration do not substitute for this decision. * Back up the existing PostgreSQL database; rehearse restoring **representative historical data** into a disposable database and upgrading it through the current Alembic head. Check ownership, version pointers, private review provenance, and file hashes before and after. Never trial migrations on the live database first. Apply PostgreSQL and ClickHouse migrations in the normal deployment sequence. * Serve the browser over HTTPS (localhost is exempt). Browser folder editing and binary downloads verify SHA-256 with Web Crypto and fail closed on insecure LAN HTTP origins rather than returning unverified bytes. * Schedule a maintenance window: turn the gate off, then drain and stop *every* old API and worker process **before running the new init container or Alembic upgrade**, not merely before enabling delivery. A normal `make rebuild` or `docker compose up` may run migrations while old services are still serving; neither Compose dependencies nor an off gate serialize schema changes with old processes. Keep the old processes stopped if initialization fails. After the migration succeeds, start only compatible server, worker and web images. Once the new API is healthy, restart `observal-lb` so nginx resolves the recreated API and web containers; an old load balancer can return 502 even when both new containers are healthy. Verify the Alembic revision and health **through the public load balancer**, and confirm its public config endpoint reports the gate off. For archive-based installs that preserve `.env`, the installer only stages the package: follow the [package cutover commands](/self-hosting/upgrades#server-package-upgrades), including selecting/pulling the new image tag, before checking health. Do not enable delivery while any old process is running. * Distribute a CLI that understands integrity-checked exact-version folder bundles, managed receipts, backups and recovery. Use HTTPS to the trusted server in production: SHA-256 hashes from that same server are not a cryptographic signature and cannot authenticate an untrusted connection. Localhost HTTP is only for isolated testing. Instruct users not to run an older CLI against a managed folder. Check all advertised skill-capable harnesses and both user/project scopes on a disposable instance. * Rehearse a separate end-to-end author → exact draft → submit → review → approved install → checked upgrade → backup restore journey on a disposable instance, including refusal to overwrite local edits and failed delivery. Review the Git/historical conversion, pinned-Agent and delegation paths separately. Preserve existing `git_fetch` and resource-less direct behavior. * If any of these checks are incomplete, leave the gate **off**, and tell authors they can only save drafts for later review. An off gate is a staging state, not an acceptable permanent developer experience for complete folders. ## Enable after the fleet is homogeneous An administrator can change **Registry → Skill folder delivery enabled** under the server's Admin Settings (the setting is stored in the DB, not an `OTEL_*` environment variable). Record the approver, deployment/client versions, migration rehearsal, and time of activation. After enabling, confirm `/api/v1/config/public` reports `skill_folder_delivery_enabled: true`; then submit a disposable draft, approve the exact candidate and verify an isolated CLI installation. Leave the gate on for normal use once these checks pass. New drafts still require an explicit submission; saving never submits or automatically approves anything. ## If the rollout fails Set the gate **off** to stop *new* complete-folder submissions and delivery while investigating. This does **not** roll back migrations, undo an existing approval, remove installed files or restore an older database. Retain receipts and backups, preserve the approved release bytes and investigate affected clients before retrying. Never reintroduce an old worker while reviews created under the new rules remain eligible. A database restore, historical downgrade or client rollback needs its own separately rehearsed recovery procedure; disabling the gate alone is not that procedure. See [the developer workflow](/skill-folder-workflow) for the user-facing lifecycle and [self-hosting upgrades](/self-hosting/upgrades) for the general deployment order. # Complete skill folders: developer workflow Source: https://docs.observal.io/skill-folder-workflow > In a fully rolled-out deployment, authors do not manage a feature switch: > save a draft, explicitly submit it for review, then install the approved > release. If this deployment has not enabled complete-folder delivery yet, > authors can save drafts but cannot finish review or installation. Ask the > operator to complete the [one-time rollout](/skill-folder-rollout); do not > turn the gate on as part of authoring. A complete skill contains a UTF-8 `SKILL.md` at its root and optional scripts, assets, and templates. The current limits are **128 extra files**, **2 MiB per file**, and **4 MiB decoded bytes for the whole folder**. Browser uploads do not retain POSIX executable bits: inspect and set executable switches before saving. For large or nested projects, use the CLI so paths and executable modes are captured directly. Symlinks, special files, hidden files and Git metadata are not included automatically; inspect the reported exclusions before proceeding. ## Create, inspect, and review ```sh theme={null} observal registry skill submit --from-dir ./my-skill --draft # Note the returned listing ID and exact draft version UUID. observal registry skill replace-files LISTING --version-id DRAFT_UUID \ --revision OBSERVED_REVISION --from-dir ./my-skill observal registry skill submit --submit LISTING --version-id DRAFT_UUID ``` The Components browser has the equivalent folder editor. Save the exact version and reopen it from **My submissions**, the listing's **Versions** tab, or its **Edit** tab; the version number and short UUID identify which draft is being edited. The owner can submit from the draft detail page or select an exact version in **My submissions**. If folder review is disabled, the draft stays editable and the submission control explains why it is unavailable. A reviewer must inspect that exact candidate before approving it. Until approval, old released bytes and their installations remain unchanged. Reviewers can use the browser or the CLI. Every file of a folder skill, including resource files, appears in the review snapshot, so the approval is bound to the exact revision the reviewer inspected. Approving never publishes by itself, and a folder with resources is never published automatically: ```sh theme={null} observal review show NUMBER observal review files NUMBER observal review diff NUMBER --file SKILL.md observal review approve NUMBER observal review publish NUMBER ``` To release a successor under the *same listing*: ```sh theme={null} observal registry skill fork NAMESPACE/SLUG --version 1.1.0 \ --description 'New templates' --from-dir ./my-skill # The command reports the saved version UUID; submit that exact UUID. observal registry skill submit --submit NAMESPACE/SLUG --version-id DRAFT_UUID ``` The owner can also use **Create next version** on an approved skill's detail page, edit the copied folder, and submit its exact draft. For a historically invalid direct release or an approved Git release, use the distinct conversion: ```sh theme={null} observal registry skill import-folder NAMESPACE/SLUG --from-dir ./my-skill \ --version 1.1.0 --description 'First complete registry folder' ``` The browser's **Import folder as next version** uses the same atomic endpoint. Supply the *entire* directory yourself. Observal does **not** fetch private Git contents for a purported file-by-file diff: reviewers see old Git source metadata and the complete new candidate. The old Git release remains installable until its successor passes review. ## Inspect and install an approved version Choose the visible approved version in the browser. The install command pins that exact version by default. **Follow latest** is an explicit, unpinned alternative. **User (global)** installs for the current user's harness; **Project** operates relative to the terminal's current project directory. Only server-advertised, skill-capable harnesses are offered. The viewer shows inert text and byte-verified binary downloads, never executes uploaded files. ```sh theme={null} observal registry skill show NAMESPACE/SLUG --version 1.1.0 observal registry skill export NAMESPACE/SLUG ./inspected-copy --version-id APPROVED_UUID observal registry skill install NAMESPACE/SLUG --version 1.1.0 --harness pi --scope project ``` `show` lists paths, byte sizes and modes without dumping their content; `export` refuses an existing destination. Project installs must be run from the intended project root. Git-backed and historical resource-less skills keep their existing install behavior while the folder-delivery gate is disabled. ## Verified managed update and recovery An installation receipt records exact registry, listing, release, target, owner, and sorted file hashes/modes in `~/.observal/lockfile.json`; it stores no file content or credentials. The CLI checks every local file, including unexpected files, before treating an existing folder as managed. Managed folder installs, upgrades, backup restores, and pinned Agent folder pulls currently require POSIX filesystem semantics (Linux/macOS or WSL on its Linux filesystem); native Windows is not supported and fails closed. This restriction does not apply to the existing Git and single-file workflows. A modified, symlinked, differently owned or ambiguous destination is **not** overwritten. Do not delete it or use `--force`; export the reviewed release, back up local changes separately, and resolve the mismatch deliberately. A historical folder without a receipt can be adopted only when its unambiguous machine lock entry and *complete* reviewed old bundle match the tree exactly. Installing an older Git or resource-less release into the same managed listing/scope is refused, even when it uses a different folder name: that transition needs an explicit manual migration rather than losing the receipt. `--check-upgrade` and `--no-write` do not record downloads on a server that supports previews. ```sh theme={null} observal registry skill install NAMESPACE/SLUG --harness pi --scope project \ --version 1.1.0 --check-upgrade observal registry skill install NAMESPACE/SLUG --harness pi --scope project \ --version 1.1.0 --upgrade observal registry skill backups list observal registry skill backups restore BACKUP_ID observal registry skill backups prune BACKUP_ID ``` A first install needs no backup, so it works whatever filesystem the default private `~/.observal/backups/skills/` is on. Replacing an existing folder moves it into the backup root by rename, so choose an explicit `--backup-root DIR` on the target filesystem if the default is on another one. `--check-upgrade` reports `added`, `removed`, `modified` (content changed) and `mode_changed` paths. The backup root must be outside all harness skill-discovery roots and ignored by Git when inside a repository. Backups are retained until explicitly pruned; restore refuses an edited active folder. A downgrade requires both an exact older `--version` and explicit update intent. A process crash can briefly leave an active path absent; the next managed invocation inspects its recovery marker and restores a byte-verified prior copy or stops with its paths for manual inspection. Do not run an old CLI against a managed folder during a mixed- version rollout. Agent pulls remain pinned by the project `observal.lock` and change versions only with an explicit `observal agent pull --upgrade` or `--version`. A legacy Agent pull or config write cannot replace an existing verified skill folder; a failed managed pull restores its previous machine lock entry after the verified folders are rolled back. Delegation's disposable worktrees are not managed installations. Human UI review, historical migration rehearsal, and mixed-version rollout verification are release gates **after** workflow testing. Completing these instructions does not authorize enabling folder delivery or merging a PR. # Testing Guide Source: https://docs.observal.io/testing/Testing_Guide This guide defines the pattern for new Python tests in Observal. The current test suite has several styles because it grew over time. Do not rewrite old tests only for style. New tests and touched files should move toward this pattern when it keeps the diff focused. ## Goals Good tests in this repo should be: * **Hermetic**: no real network, Docker, user config, or external services. * **Behavioral**: assert user-visible behavior or service contracts, not incidental internals. * **Small**: one route group, command, service, or behavior area per file. * **Explicit**: setup is visible in the test or in small local helpers. * **Fast**: sleeps, external calls, and expensive setup are mocked at the boundary. ## File layout Use this structure for most new test files: ```python theme={null} # SPDX-FileCopyrightText: 2026 Your Name # SPDX-License-Identifier: Apache-2.0 """Tests for the behavior under test. Cover the most important scenarios in a short list when useful: * successful path * denied access * edge case or regression """ from __future__ import annotations import uuid from unittest.mock import MagicMock def _make_item(*, name: str = "example") -> MagicMock: item = MagicMock() item.id = uuid.uuid4() item.name = name return item class TestExampleBehavior: """Tests for the public behavior of ExampleBehavior.""" def test_returns_name_for_valid_item(self) -> None: from services.example import item_name item = _make_item(name="alpha") result = item_name(item) assert result == "alpha" ``` Keep the order consistent: 1. SPDX header 2. Module docstring 3. `from __future__ import annotations` when useful 4. Standard library imports 5. Test library imports 6. App imports needed for the test harness 7. Local helper factories 8. Test classes or test functions ## File scope Prefer one file per behavior area: * `tests/test_agent_name_lookup.py` for agent lookup behavior * `tests/test_component_versions_api.py` for component version endpoints * `observal_cli/tests/test_cmd_scan.py` for the `scan` command * `observal-server/tests/test_jwt.py` for JWT service behavior Split a file when it mixes unrelated layers. For example, schema validation, route behavior, config generation, and resolver service behavior should usually live in separate files. ## Imports Use top-level imports for the test harness: * Standard library modules * `pytest` * `unittest.mock` * Test clients such as `AsyncClient`, `ASGITransport`, and `CliRunner` * Models, schemas, routers, and dependency functions needed by local helpers Inline imports are useful for the unit under test: ```python theme={null} def test_resolves_agent_components() -> None: from services.agent_resolver import ResolvedAgent resolved = ResolvedAgent( agent_id=uuid.uuid4(), agent_name="demo", agent_version="1.0.0", ) assert resolved.ok is True ``` Use inline imports when importing the unit under test needs environment setup, has side effects, or makes the test easier to isolate. Do not force every import to be inline. ## Helpers and fixtures Prefer small helper functions for local setup: ```python theme={null} def _make_user(*, role: UserRole = UserRole.user) -> User: user = MagicMock(spec=User) user.id = uuid.uuid4() user.role = role user.email = "test@example.com" user.username = "testuser" return user def _mock_db() -> AsyncMock: db = AsyncMock() db.add = MagicMock() db.commit = AsyncMock() db.refresh = AsyncMock() db.flush = AsyncMock() return db ``` Use `pytest.fixture` when pytest lifecycle behavior is helpful: * Temporary directories through `tmp_path` * Environment isolation through `monkeypatch` * Cleanup with `yield` * Shared setup used by many files through `conftest.py` Prefer helpers over fixtures when the setup takes parameters or is only used in one file. ## Test body shape Use a simple arrange, act, assert flow with blank lines between phases: ```python theme={null} def test_owner_can_view_pending_listing() -> None: user = _make_user(role=UserRole.user) listing = _make_listing(status=ListingStatus.pending, created_by=user.id) result = can_view_listing(listing, user) assert result is True ``` Comments are only needed when the reason is not obvious. A clear test name is better than a comment that repeats the code. ## Naming Use behavior names: ```python theme={null} def test_missing_auth_returns_401() -> None: def test_owner_can_install_pending_listing() -> None: def test_scan_does_not_modify_ide_files() -> None: def test_hyphenated_name_resolves_by_name() -> None: ``` Avoid vague names: ```python theme={null} def test_case_1() -> None: def test_function_works() -> None: def test_route() -> None: ``` A good pattern is: ```text theme={null} test__ ``` Inside a class, the class can provide the subject: ```python theme={null} class TestAgentLookup: def test_hyphenated_name_returns_agent(self) -> None: ... ``` Use a test class when there are at least a few related tests. Module-level test functions are fine for one or two simple cases. ## Mocking Mock boundaries, not the behavior under test. Good boundaries to mock: * HTTP clients and webhooks * Database session methods * Filesystem writes when the write itself is not under test * Sleep and time delays * Auth providers * External CLIs and subprocesses Avoid mocking: * Pydantic schemas * Small pure functions * The function being tested * Several layers of internal implementation at once Use `AsyncMock` for async methods and `MagicMock` for sync methods. Use `spec` when mocking model objects: ```python theme={null} user = MagicMock(spec=User) ``` Keep patches close to the test that needs them: ```python theme={null} @patch("api.routes.agent.crud._load_agent") async def test_name_lookup_returns_agent(mock_load: MagicMock) -> None: ... ``` ## Async tests Mark async tests explicitly: ```python theme={null} @pytest.mark.asyncio async def test_fetches_current_user() -> None: from services.users import fetch_current_user db = _mock_db() user = await fetch_current_user(db, user_id=uuid.uuid4()) assert user is not None ``` The root pytest config sets asyncio mode to auto, but the marker keeps async tests easy to spot. ## API route tests Use a small FastAPI app with dependency overrides. Do not boot the full application unless the integration boundary is the thing being tested. ```python theme={null} import uuid from unittest.mock import AsyncMock, MagicMock import pytest from fastapi import FastAPI from httpx import ASGITransport, AsyncClient from api.deps import get_current_user, get_db from api.routes.agent import router from models.user import User, UserRole def _make_user() -> User: user = MagicMock(spec=User) user.id = uuid.uuid4() user.role = UserRole.user user.email = "test@example.com" user.username = "testuser" return user def _mock_db() -> AsyncMock: db = AsyncMock() db.execute = AsyncMock() db.commit = AsyncMock() return db def _app_with( *, user: User | None = None, db: AsyncMock | None = None, ) -> tuple[FastAPI, AsyncMock, User]: user = user or _make_user() db = db or _mock_db() app = FastAPI() app.include_router(router) app.dependency_overrides[get_current_user] = lambda: user app.dependency_overrides[get_db] = lambda: db return app, db, user class TestAgentLookup: """Tests for agent lookup routes.""" @pytest.mark.asyncio async def test_unknown_agent_returns_404(self) -> None: app, db, _ = _app_with() db.execute.return_value = MagicMock(one_or_none=MagicMock(return_value=None)) async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: response = await client.get("/api/v1/agents/missing-agent") assert response.status_code == 404 ``` For route tests, assert the response status and response body first. Assert database calls only when they are part of the contract. ## CLI tests CLI tests should not read or write the real home directory. Redirect home and current working directory to a temp path: ```python theme={null} from __future__ import annotations import json from typing import TYPE_CHECKING import pytest from typer.testing import CliRunner from observal_cli.main import app if TYPE_CHECKING: from pathlib import Path runner = CliRunner() @pytest.fixture() def sandbox_home(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: monkeypatch.setenv("HOME", str(tmp_path)) monkeypatch.setenv("USERPROFILE", str(tmp_path)) monkeypatch.chdir(tmp_path) return tmp_path def _write_json(path: Path, data: dict) -> None: path.parent.mkdir(parents=True, exist_ok=True) path.write_text(json.dumps(data, indent=2)) class TestScanCommand: """Tests for the scan command.""" def test_empty_home_exits_with_message(self, sandbox_home: Path) -> None: result = runner.invoke(app, ["scan"]) assert result.exit_code == 1, result.output assert "No harness configurations found" in result.output ``` Always include `result.output` in exit code assertions so failures are easy to diagnose. ## Parametrize Use parametrization for the same behavior across multiple inputs: ```python theme={null} @pytest.mark.parametrize( ("role", "expected"), [ (UserRole.admin, True), (UserRole.reviewer, True), (UserRole.user, False), ], ) def test_role_can_review(role: UserRole, expected: bool) -> None: result = can_review(role) assert result is expected ``` Do not use a large parameter matrix when separate named tests would be easier to read. ## Property-based tests `hypothesis` is available for invariants that should hold over many inputs. Use it for pure logic, parsers, serializers, redaction, and validation edge cases. ```python theme={null} from hypothesis import given, strategies as st class TestRedactionProperties: """Property tests for support bundle redaction.""" @given(secret=st.text(min_size=1, max_size=100)) def test_secret_value_is_removed(self, secret: str) -> None: from observal_cli.support.redaction import redact_text result = redact_text(f"token={secret}", secrets=[secret]) assert secret not in result ``` Keep property-based tests deterministic and focused. If a failing example reveals a bug, add a named regression test too. ## Fuzzing Property tests cover invariants you can name. Fuzzing covers the ones you cannot: the Atheris targets live in `fuzz/`, and the project is being submitted to [Google OSS-Fuzz](https://google.github.io/oss-fuzz/) for continuous fuzzing. Reach for a fuzz target instead of a property test when the boundary takes untrusted bytes and the interesting failures are crashes rather than wrong answers. Session transcripts, the ingest classifiers, and both redaction layers are covered today. ```bash theme={null} make test-fuzz # smoke-test every target python3 fuzz/session_jsonl_fuzzer.py -atheris_runs=100000 python3 fuzz/session_jsonl_fuzzer.py crash- # reproduce a finding ``` `fuzz/README.md` covers adding a target, seed corpora, dictionaries, and the OSS-Fuzz project configuration. ## Assertions Use plain `assert`. Assert the public result before internal interactions: ```python theme={null} assert response.status_code == 200 assert response.json()["name"] == "my-agent" ``` Use call assertions when the call is the behavior: ```python theme={null} mock_post.assert_called_once_with("/ingest", json=payload) ``` For CLI tests, include output in the assertion message: ```python theme={null} assert result.exit_code == 0, result.output ``` For exceptions, assert the important fields: ```python theme={null} with pytest.raises(HTTPException) as exc_info: await require_admin(user) assert exc_info.value.status_code == 403 assert "admin" in exc_info.value.detail.lower() ``` ## Test directories Use the existing directories: * `tests/` for cross-cutting backend, CLI, and integration-style unit tests * `observal-server/tests/` for server-focused tests that live with the server package * `observal_cli/tests/` for CLI command and CLI package tests * `tests/e2e/` for Playwright tests that require the running stack Shared setup should stay small: * `tests/conftest.py` adds the server source path for imports. * `observal-server/tests/conftest.py` sets test environment variables and shared auth fixtures. Extend `conftest.py` only when setup is useful across several files. ## Running tests Use the make targets for normal workflows: ```bash theme={null} make test make test-v make test-eval-completeness make test-adversarial make test-fuzz make test-all ``` Run focused pytest commands from `observal-server` when iterating on one file: ```bash theme={null} cd observal-server uv run pytest ../tests/test_agent_name_lookup.py -q uv run pytest ../observal-server/tests/test_jwt.py -q uv run pytest ../observal_cli/tests/test_cmd_scan.py -q ``` Use `make lint` and `make format` before pushing Python changes. Use `make check` when you need the full pre-commit suite. ## Formatting and lint notes The root `pyproject.toml` relaxes a few lint rules for tests. Those ignores are for practical mock setup, not a reason to leave tests messy. Current per-file ignores include: ```toml theme={null} [tool.ruff.lint.per-file-ignores] "tests/*" = ["F841", "RUF059", "B007", "TID251"] "observal-server/tests/*" = ["F841", "RUF059", "B007", "TID251", "E402"] ``` In practice: * Unused mock variables are allowed in test files. * Some server tests can use guarded imports after setup. * Tests should import feature modules from their normal server package paths. * New tests should still prefer clear imports, typed helpers, and small setup. ## Review checklist Before submitting a test change, check that: * The test fails without the product fix or covers a meaningful invariant. * External services are mocked or replaced with local fakes. * Real user config and real home directories are not touched. * The test name states the behavior. * Helpers are local unless they are reused across files. * Async tests use `AsyncMock` for awaited methods. * CLI tests include command output in exit code assertions. * API route tests override dependencies in a local FastAPI app. * Focused tests pass locally. # Debug agent failures from session evidence Source: https://docs.observal.io/use-cases/debug-agent-failures AI failures rarely produce one reliable error code. Observal preserves the session transcript so you can inspect what the user asked, which tools ran, what results the harness recorded, and how the agent responded. ## 1. Find the session Open `/traces` in the web UI or list recent sessions: ```bash theme={null} observal ops traces --limit 50 ``` Filter by harness, agent, user, model, or time range. Select the session that matches the reported failure. ## 2. Inspect the event sequence Expand the session to review prompts, assistant responses, tool calls, tool results, lifecycle events, and subagent activity in source order. The CLI can unfold the same session structure: ```bash theme={null} observal ops traces --turn --limit 10 ``` Look for repeated tool calls, error results, missing inputs, unexpected model responses, or a stop event that occurred before the expected work completed. ## 3. Inspect recorded tool detail The detail available for a tool call depends on the harness transcript. When present, Observal displays the tool name, input, result, and error indicators. Observal does not intercept MCP transport traffic or invent payload fields that the harness did not record. ## 4. Common patterns | Session pattern | Likely cause | | - | - | | Same tool called repeatedly with identical arguments | The agent is stuck in a retry loop or is not consuming the result | | Tool result records an error and the assistant continues | Prompt or recovery policy did not handle the failure | | Tool call has no matching result | Harness shutdown, interrupted process, or incomplete transcript delivery | | Session stops immediately after a prompt | Hook, model, permission, or harness configuration issue | | Expected MCP call never appears | The model did not select the tool, or the harness transcript omits that detail | ## 5. Recover incomplete delivery If the local harness still contains the session source, run: ```bash theme={null} observal reconcile ``` Reconciliation resumes from the acknowledged checkpoint and replays missing source records idempotently. # Fork and customize an agent or component Source: https://docs.observal.io/use-cases/fork-and-customize Fork an **approved release** when you want to build on a registry item without editing its owner's copy. A fork is a new listing with its own owner, namespace, releases, reviews, downloads, ratings, and telemetry. Agent forks retain **pinned references** to approved components; they do not fork those components. ## In the web UI 1. Open an approved agent or component and select **Fork**. Choose an approved base version, a new name, and your namespace or an eligible teamspace. A private team source can only be forked into the same private teamspace. 2. The new item opens as your **draft**. An agent opens in the Builder; a component opens in its draft editor. Edit and save it before submitting for review. 3. On the detail page or in the review queue, select **Diff vs upstream** to compare your version with the approved base. **Unchanged from upstream** means tracked content has not changed. A version number, validation state, or a secret environment/header value alone does not count as an install-content change. 4. Submit the draft through its normal review workflow. Approval makes a public fork appear in the source's **Forks** tab and count. Drafts and team-private forks never appear there. If you later lose access to the source, provenance reads **Source unavailable**. The comparison will no longer be available; Observal does not expose a saved source name or base version as a fallback. ## From the CLI ```bash theme={null} observal agent fork acme/pr-reviewer --name my-reviewer --dir ./my-reviewer --output json # Edit ./my-reviewer/observal-agent.yaml, then update the *fork*, not the source: observal agent publish --update --dir ./my-reviewer observal agent publish --submit observal registry skill fork-listing acme/review-skill --name my-review-skill --output json # Edit and submit the resulting skill draft through its normal registry workflow. ``` The `--dir` scaffold stores the fork's ID, so subsequent `publish --update` calls target that draft rather than creating another listing. If a name is already taken, choose another; the existing item is never overwritten. For scripted integrations, use `POST /api/v1/agents/{id}/fork` or `POST /api/v1/{type}/{id}/fork` (`type` is `mcps`, `skills`, `hooks`, `prompts`, or `sandboxes`). `GET …/{id}/fork-diff?version=…` compares a visible fork version with its approved, currently accessible base. ARD search can filter currently **public and approved** direct forks using `query.filter["obs:forksOf"]` set to the upstream's `urn:air:` identifier; public results then include `obs:forkedFrom`. Private and unpublished lineage is never exposed through this facet. # Use Cases Source: https://docs.observal.io/use-cases/index Observal solves five concrete problems in the AI-agent lifecycle. Each page below is written as a playbook: what the problem is, how Observal addresses it, and the exact commands to run. | If you need to... | Read | | - | - | | See what your MCP servers are actually doing | [Observe MCP traffic](/use-cases/observe-mcp-traffic) | | Package an agent once and ship it to any harness | [Share agent configs across harnesses](/use-cases/share-agent-configs) | | Figure out why a session went wrong | [Debug agent failures](/use-cases/debug-agent-failures) | | Give your whole team a source of truth | [Run a team-wide agent registry](/use-cases/team-registry) | | Adapt an approved agent or component without changing the original | [Fork and customize](/use-cases/fork-and-customize) | ## How these relate ```mermaid theme={null} flowchart LR observe[Observe MCP traffic] debug[Debug agent failures] share[Share agent configs] registry[Team registry] observe -->|traces give you evidence| debug observe --> share share -->|governance + reuse| registry ``` You typically adopt Observal in that order: **observe** first (low-risk, instant value), then **debug**, then **share**, and finally **run a team registry** once you have enough published agents to justify governance. # Sync an MCP server from GitHub or GitLab Source: https://docs.observal.io/use-cases/mcp-repo-sync An MCP server listed with a git repository can publish new versions on its own. Add a webhook to the repository on GitHub or GitLab, and Observal re-reads the repository and publishes a new version whenever you push or publish a release. You no longer have to run `observal registry mcp edit` after every change. GitHub (github.com and GitHub Enterprise) and GitLab (gitlab.com and self-managed GitLab) are supported. ## What a sync does 1. GitHub or GitLab sends a webhook to Observal. 2. Observal authenticates it and checks the repository. Then it queues the sync and replies right away. 3. A background worker fetches the repository, runs the same analysis as `observal registry mcp submit --git`, and publishes a new version. Each sync creates a new version and leaves earlier versions alone. Agents pinned to an earlier version keep installing it, because agent lockfiles pin version content by digest. The new version starts as a copy of the current version, then picks up these changes from the repository: | Field | What changes | | - | - | | Tools | Replaced with the tools found in the repository | | Environment variables | Newly found variables are added as optional. Variables you already listed keep your descriptions | | Framework | Updated when one is detected | | Command, args, Docker image, setup instructions | Filled in only when the listing has none, and only from an image the repository declares in a compose file or README. A guessed image name is never stored. Values you set by hand are kept | | Description | Kept | | Commit and ref | The synced commit sha and the branch or tag name are recorded on the version | | Changelog | The head commit message for a push, or the release notes for a release | Synced versions are approved and published right away, without review. Anyone who can push to the tracked branch, or publish a release, can change what the listing installs. Turn on sync only for repositories whose write access you trust. ## Triggers and version numbers Choose either trigger, or both: | Trigger | Fires on | Version | | - | - | - | | Push | A push to the tracked branch. The default is the repository's default branch | The version in `pyproject.toml` or `package.json` when it is higher than every existing version. Otherwise the highest existing version with its patch number raised, so `1.2.3` becomes `1.2.4` | | Release | A GitHub release being published (drafts are ignored), or a GitLab release being created | The release tag, either `v1.2.3` or `1.2.3`. A tag that is not a semantic version is ignored | Pushing a tag on its own does not sync. Create a release from the tag. If fetching the repository fails, the sync retries twice, 15 and then 30 seconds later, before it is marked failed. This covers network errors and a release whose tag is still being published. A push sync always fetches the branch tip when the job runs, not the commit named in the delivery. Several quick pushes therefore publish the newest code, and a commit that is already published is skipped. If an earlier sync finishes after a later one, it is skipped too, so older code never becomes the latest version. A release whose version already exists is skipped too. A release older than the current latest version is published but does not become the latest. ## Set it up You must own the listing, or be a co-author or an admin. The listing must have a git repository URL, and a reviewer must have approved it at least once. Synced versions skip review, so the first version still goes through it. ### 1. Turn on sync From the CLI: ```bash theme={null} observal registry mcp sync enable alice/weather-mcp # push only observal registry mcp sync enable alice/weather-mcp --release # push and release observal registry mcp sync enable alice/weather-mcp --release --no-push --branch stable observal registry mcp sync enable alice/weather-mcp --provider gitlab # self-hosted GitLab ``` Or open the MCP server in the web app and use the **Sync** tab. Observal picks the provider from the repository URL: a hostname containing `gitlab` means GitLab, anything else means GitHub. For a self-managed GitLab on another hostname, such as `code.acme.com`, choose GitLab with `--provider gitlab` or the **Repository host** setting in the Sync tab. Observal returns a webhook URL and a secret. The secret is shown only once. If you lose it, rotate it with `observal registry mcp sync rotate-secret`. ### 2. Add the webhook #### GitHub In the repository, open **Settings > Webhooks > Add webhook** and enter: | Setting | Value | | - | - | | Payload URL | The webhook URL from step 1 | | Content type | `application/json` | | Secret | The secret from step 1 | | Events | **Just the push event** for push sync. For release sync, choose **Let me select individual events** and tick **Releases**, plus **Pushes** if you use both | GitHub sends a `ping` right away. Its delivery should show a `202` response. #### GitLab In the project, open **Settings > Webhooks > Add new webhook** and enter: | Setting | Value | | - | - | | URL | The webhook URL from step 1 | | Secret token | The secret from step 1 | | Trigger | **Push events** for push sync, **Releases events** for release sync, or both. You can limit push events to the tracked branch, though Observal ignores other branches anyway | | SSL verification | Keep it on | GitLab has no ping event. Use **Test > Push events** on the saved webhook to send a sample push. It syncs the tracked branch, or is skipped if that commit is already published. The provider must be able to reach your Observal server. Set `deployment.public_url` so the webhook URL uses your public address. If Observal runs on a private network, see [Private deployments](#private-deployments). ### 3. Check the result ```bash theme={null} observal registry mcp sync status alice/weather-mcp observal registry mcp sync run alice/weather-mcp # sync the tracked branch now ``` Deliveries that are ignored, such as a push to another branch, do not change the sync status. The provider's delivery log (**Recent Deliveries** on GitHub, **Recent events** on GitLab) shows the reason in the response body. ## Private deployments github.com and gitlab.com deliver webhooks over the public internet, so an Observal install inside a private network needs one public path for them. Expose only the webhook receivers, `POST /api/v1/webhooks/github/mcp/` and `POST /api/v1/webhooks/gitlab/mcp/`, through a public gateway, load balancer or reverse proxy that answers `404` to every other path. Each delivery is authenticated by its signature or secret token, so a source IP allowlist is optional. Then set `WEBHOOK_PUBLIC_URL` on the API to that public address, for example `https://hooks.observal.example.com`. The **Sync** tab shows webhook URLs under it, while everything else keeps using the private address. * **AWS Terraform:** set `enable_webhook_ingress = true` and list your providers in `webhook_providers` to add an API Gateway endpoint that does this. See [Git webhooks on a private install](/self-hosting/aws-terraform#git-webhooks-on-a-private-install). * **GitHub Enterprise Server or self-managed GitLab** on the same network can reach Observal directly and needs none of this. Self-managed GitLab blocks webhooks to private addresses by default; an administrator allows them under **Admin > Settings > Network > Outbound requests**. If GitLab reports `URL is blocked: Host cannot be resolved or invalid`, the GitLab server cannot resolve Observal's hostname. ## Private repositories Observal fetches the repository with the server's `GIT_CLONE_TOKEN`, the same token used to analyze git submissions. The token needs read access to the repository. Set `GIT_CLONE_TOKEN_HOSTS` to the exact HTTPS hostname of your Git provider (`github.com` or your GitLab host); without it, the token is never sent. For GitLab, set `GIT_CLONE_TOKEN_USER=oauth2`. A git server on a private network also needs `ALLOW_INTERNAL_GIT_URLS=true`. ## Security * Each listing has its own secret, stored encrypted. GitHub deliveries need a valid `X-Hub-Signature-256` signature and GitLab deliveries the matching `X-Gitlab-Token`; anything else is rejected with `401`. * Each listing answers only on its own provider's endpoint, so a GitLab secret is never accepted as a GitHub signature or the reverse. * A delivery from a different repository than the listing's git URL is rejected with `422`. * Synced versions are published as the user who last turned on or changed sync. If that user no longer has owner-level access to the listing (as its owner, a co-author or an admin), or was removed from the teamspace of a private listing, syncs fail until an owner turns sync off and on again. * The receivers allow 60 deliveries per minute per client address. ## Why sync is one way Sync only reads from the repository; editing the listing in Observal never writes back to GitHub or GitLab. The repository is the source of truth for the server's code, and the fields Observal derives from it (tools, environment variables, version) are a read of that code. Writing back would need a token that can push to every synced repository, could loop (a push syncs, the sync pushes), and would race with the repository's own history. What you edit in Observal, such as the description or environment variable descriptions, is listing metadata and is kept across syncs. ## Turn it off ```bash theme={null} observal registry mcp sync disable alice/weather-mcp ``` Published versions stay. Remove the webhook from the repository's settings as well. # Observe MCP activity in sessions Source: https://docs.observal.io/use-cases/observe-mcp-traffic Observal identifies MCP and tool activity from the session transcript recorded by each coding harness. It does not intercept MCP transport traffic. ## What you get When the harness records the relevant detail, session events can include: * Tool or MCP name * Tool input and result * Event ordering within the session * Harness, user, agent, and model attribution * Token totals and timing derived from the session Coverage varies by harness because Observal can only index fields present in the local transcript or native message API. Transport-level latency and payload data are not synthesized when the harness does not record them. ## Discover and install session hooks ```bash theme={null} observal scan observal doctor patch --all-harnesses ``` `scan` is read-only. `doctor patch` installs supported session hooks or extensions and leaves every MCP command and remote URL unchanged. Scope hook installation to selected harnesses: ```bash theme={null} observal doctor patch --harness claude-code observal doctor patch --harness kiro observal doctor patch --harness copilot-cli ``` Restart the harness after hook changes, then run a coding session. If a hook was missed or the machine was offline, reconcile local session sources: ```bash theme={null} observal reconcile ``` ## Query collected sessions Open `/traces` in the web UI to filter sessions by harness, agent, user, model, and time range. Expand a session to inspect parsed prompts, responses, tool calls, tool results, and lifecycle events. The CLI can list recent sessions and unfold their events: ```bash theme={null} observal ops traces --limit 20 observal ops traces --turn --limit 10 ``` ## Caveats * MCP visibility depends on the harness transcript format. * Observal does not modify, proxy, or wrap MCP traffic. * Monetary cost and structured transport errors are unavailable unless the harness records equivalent session fields. ## Next [Debug agent failures](/use-cases/debug-agent-failures) explains how to investigate failures using session evidence. # Share agent configs across harnesses Source: https://docs.observal.io/use-cases/share-agent-configs Your team has a reviewer agent that works great in Claude Code. Now someone wants to use it in Kiro. Then Cursor. Then they want to tweak the skills. Copy-pasting config snippets across tools doesn't scale. Observal's registry gives you one agent definition that installs cleanly into every supported harness. ## The shape of an agent Every agent is a YAML file that bundles: * MCP servers it needs * Skills to load * Hooks to wire into the session lifecycle * Prompts (with variables) * Sandboxes for code execution When someone runs `observal agent pull `, Observal templates that YAML into the right files for their harness: `~/.claude/agents/*.json`, `.kiro/agents/*.json`, `.cursor/mcp.json`, and so on. ## Publish an agent ### Option A - the interactive wizard ```bash theme={null} observal agent create ``` Step-by-step prompts: name, description, which MCP servers, which skills, which hooks. Results in a registry entry you can share by ID. ### Option B - the YAML workflow (recommended for teams) ```bash theme={null} observal agent init # scaffold observal-agent.yaml observal agent add mcp github-mcp # add components observal agent add skill code-review-skill observal agent add hook pretooluse-logger observal agent build # validate (dry-run) observal agent publish # submit to registry ``` The YAML workflow is PR-reviewable. The file lives in your repo; changes flow through your normal review process. ## Install an agent into any harness Browse what exists: ```bash theme={null} observal agent list observal agent list --search review observal agent show ``` Install with one command, pick the harness: ```bash theme={null} observal agent pull --harness claude-code observal agent pull --harness kiro observal agent pull --harness cursor observal agent pull --harness vscode observal agent pull --harness codex ``` The CLI prompts for any environment variables the MCP servers declare as required (GitHub tokens, API keys). These are stored in your harness config, not uploaded to Observal. ### Control what gets installed ```bash theme={null} # Preview without writing anything observal agent pull --harness claude-code --dry-run # Install into a specific directory observal agent pull --harness claude-code --dir ./my-project # Claude Code only: scope (project-local vs user-global) observal agent pull --harness claude-code --scope project observal agent pull --harness claude-code --scope user # Claude Code only: sub-agent model observal agent pull --harness claude-code --model sonnet # Claude Code only: tool allowlist observal agent pull --harness claude-code --tools Read,Write,Bash ``` ## What portability actually means The harness feature matrix (defined in `packages/observal-shared/observal_shared/harness_registry.py`) controls what each harness supports. If an agent uses skills and the target harness doesn't have skills, the installer: * Installs the compatible parts cleanly * Warns about the unsupported parts * Exits non-zero if the agent *requires* something the harness cannot provide Useful when onboarding a new machine or swapping between "work setup" and "personal setup." ## Next → [Run a team-wide agent registry](/use-cases/team-registry): once publishing is routine, you need governance. # Run a team-wide agent registry Source: https://docs.observal.io/use-cases/team-registry Once two or more people are authoring agents, you need a single source of truth. Observal becomes your team's internal Docker Hub for AI agents, with review, RBAC, and telemetry baked in. ## What changes at team scale * **Discovery**: everyone sees the same list of agents, MCPs, skills, hooks, prompts, and sandboxes. * **Review**: admins approve what appears in the public listing. Authors' own items are still immediately usable. * **Governance**: RBAC roles (`super_admin`, `admin`, `reviewer`, `user`) control who can publish and approve. * **Visibility**: centralized dashboards instead of "ask Sarah which version she's running." ## Setup shape Deploy once, everyone points at it. ```mermaid theme={null} flowchart TB server["Observal server - API + Web UI"] db[(PostgreSQL)] ch[(ClickHouse)] redis[(Redis)] a["Engineer A - Claude Code"] b["Engineer B - Kiro"] c["Engineer C - Cursor"] server --> db server --> ch server --> redis a --> server b --> server c --> server ``` Install the server once ([Self-Hosting](/self-hosting)). Then every engineer installs the CLI and runs `observal auth login` pointed at your shared server URL. ## Users and roles Four roles, RBAC-enforced on every endpoint. | Role | Can | Cannot | | - | - | - | | `user` | Publish components (subject to review), install agents, view their own traces | Approve submissions, see other users' private traces, change server settings | | `reviewer` | Everything `user` can + approve/reject submissions | Change server settings, manage users | | `admin` | Everything `reviewer` can + manage users, change server settings | Only restriction: certain super-admin operations | | `super_admin` | Everything | - | Manage users: ```bash theme={null} observal admin users --output json observal admin reset-password --generate --output json observal admin delete-user --force --output json ``` Change a role via the web UI (`/settings/users`) or the API (`PUT /api/v1/admin/users/{id}/role`). ## Onboarding a new engineer Two commands to get them productive: ```bash theme={null} # The new engineer runs: curl -fsSL https://raw.githubusercontent.com/Observal/Observal/main/install.sh | bash observal auth login --server https://observal.your-company.internal ``` For managed deployments, users authenticate through SSO or are provisioned by an admin. See [Authentication and SSO](/self-hosting/authentication). After logging in, they can: ```bash theme={null} observal agent list # see every agent the team has published observal agent pull team-reviewer --harness claude-code # install one observal scan # discover what they have installed observal doctor patch --all-harnesses # instrument everything ``` ## Review workflow Authors submit. Reviewers approve. Approved items appear in the public listing. ```bash theme={null} observal review list --output json observal review show --output json observal review approve --output json observal review request-changes --body "missing env var docs" --output json observal review publish --output json ``` What reviewers look for: * Does the README/description make it clear what the component does? * Does the MCP analysis (from `submit`) look correct: tools, env vars, transport? * Are required env vars documented? * Is the repo URL trustworthy (pinned commit or tag)? Everything published is visible to the author immediately. Review controls what appears in the public listing. ## Telemetry across the whole team Because every engineer's session telemetry flows into the same server, `observal ops` becomes a team dashboard: ```bash theme={null} observal ops top --type agent # most-used agents across the team observal ops top --type mcp # hottest MCP servers ``` Filters in the web UI let you slice by user, agent, harness, and time range. ## SSO and audit considerations For orgs that need SSO and audit logging, configure OIDC in **Admin → SSO**: | Setting | Value | | - | - | | `oauth.client_id` | Your IdP client ID | | `oauth.client_secret` | Your IdP client secret | | `oauth.server_metadata_url` | Your IdP discovery URL | Restart the API after OIDC changes. See [Authentication and SSO](/self-hosting/authentication). ## Next → [Self-Hosting](/self-hosting): the operator's playbook for actually running the server this use case depends on. # Teamspaces Source: https://docs.observal.io/use-cases/teamspaces A teamspace is a shared registry namespace with a unique handle such as `platform-tools`. Team handles and usernames cannot collide. ## Roles Every member has one role: | Role | Can | | - | - | | `owner` | Manage members, invites, visibility, and registry content | | `reviewer` | Approve or reject the teamspace's pending agents and components | | `member` | View the roster and publish into the teamspace | Global admins can manage any teamspace. A teamspace always keeps at least one owner. ## Creating a teamspace Any signed-in user can create a teamspace under **Registry, Teamspaces**. Private teamspaces are ready immediately. A public teamspace reserves its handle but stays private and locked until a global reviewer or above approves it in the Review queue. A global reviewer may approve their own request. Changing an existing private teamspace to public follows the same review flow. A rejected request keeps the teamspace private and can be submitted again. Personal teamspaces always remain private. ## Publishing and review Private teamspaces can publish only team-private agents and components. Teamspace submissions never auto-approve. After a teamspace is approved for public visibility, its owners and team reviewers may approve or reject all content in that namespace, including public content and their own submissions. Global reviewers continue to review public content across the registry. Public submissions notify both groups, while team-private submissions notify only that teamspace's owners and reviewers. Global admins may still review all content. A public teamspace cannot become private while it owns public agents, components, or component sources. Restrict or remove those items first. ## Membership Owners and global admins can add users and assign roles. Public teamspaces accept join requests. Private teamspaces use expiring invite links, and their owners decide the resulting requests. Members can leave unless they are the last owner.