AWS deployment with Terraform

What gets provisioned

• VPC with public + private subnets across two availability zones, NAT gateway, VPC flow logs
• Application Load Balancer with HTTPS (ACM certificate, DNS-validated) when you supply a domain; HTTP-only otherwise. Path-based rules: /api/* → api service, optional /grafana/* → Grafana, default → web
• ECS Fargate cluster running:
  • api (FastAPI): 2 tasks by default, autoscales 2–10 on CPU
  • web (Vite): 2 tasks by default, autoscales 2–6 on CPU
  • worker (arq background jobs): 1 task by default, autoscales 1–5 on CPU
  • init (one-shot migrations + seeds): runs as a Fargate RunTask whenever image_tag changes
• RDS Postgres 16: Multi-AZ on prod, encrypted, automated daily backups, Performance Insights, Enhanced Monitoring, log exports
• ElastiCache Redis 7: 2-node replication group with automatic failover on prod, slow-log to CloudWatch
• Data tier EC2 (Amazon Linux 2023): single host running ClickHouse on EBS gp3, optional Prometheus and Grafana, ENI with static private IP, internal Route 53 zone for DNS, daily ClickHouse → S3 snapshot via systemd timer
• S3 backups bucket: versioned, AES256, lifecycle to STANDARD_IA → GLACIER_IR → expire, TLS-only
• CloudWatch log groups: per ECS service, data host, RDS, Redis slow log, VPC flow logs
• SSM Parameter Store: generated DB / ClickHouse / SECRET_KEY / optional Grafana passwords, plus pre-built connection URLs injected into ECS tasks
• SSM Session Manager: shell access to the data host, no SSH

Prerequisites

Quickstart

git clone https://github.com/Observal/Observal.git
cd Observal/infra/terraform/aws

# 1. Authenticate to AWS (any of these works)
export AWS_PROFILE=observal-prod
# or: aws configure sso
# or: export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...

# 2. Configure
cp terraform.tfvars.example terraform.tfvars
$EDITOR terraform.tfvars

# 3. Apply
terraform init
terraform plan -out tf.plan
terraform apply tf.plan

terraform output app_url

Configuration

Minimal configuration (HTTP, no custom domain)

region      = "us-east-1"
environment = "prod"

Recommended configuration (HTTPS on your domain)

region      = "us-east-1"
environment = "prod"
sizing      = "medium"

domain_name     = "observal.example.com"
route53_zone_id = "Z0123456789ABCDEFGHIJ"

# Lock down ingress to your corporate egress / VPN
alb_ingress_cidrs = ["203.0.113.0/24", "198.51.100.0/24"]

Sizing presets

sizing = "medium"   # small | medium | large | custom

• When sizing = "small|medium|large": all individual resource variables are ignored. The preset values are used unconditionally.
• When sizing = "custom": individual variables take effect normally.

Custom sizing

sizing = "custom"

api_cpu              = 512   # 0.5 vCPU; raise to 1024+ for sustained >100 req/s
api_memory           = 1024
api_desired_count    = 2
api_autoscale_max    = 10

web_desired_count    = 2
worker_desired_count = 1

# Data tier (ClickHouse)
data_instance_type   = "t3.large"   # 8 GB - minimum viable for ClickHouse
data_volume_size_gb  = 100
observability_stack  = "none"       # none | prometheus | grafana

db_instance_class    = "db.t4g.small"
redis_node_type      = "cache.t4g.micro"

Application options

deployment_mode     = "enterprise"   # SSO-only login
data_retention_days = 90             # ClickHouse TTL; 0 to disable
log_retention_days  = 30             # CloudWatch log group retention
image_tag           = "v1.4.0"       # specific Observal release; "latest" pulls main

Bring Your Own VPC

# Use an existing VPC
vpc_id             = "vpc-0abc1234def56789a"
private_subnet_ids = ["subnet-aaa11111", "subnet-bbb22222"]
public_subnet_ids  = ["subnet-xxx11111", "subnet-yyy22222"]

# Control ALB placement independently
alb_scheme = "internet-facing"   # or "internal"

• DNS support enabled (enableDnsSupport = true, enableDnsHostnames = true)
• Private subnets must have outbound internet access (via NAT Gateway or Transit Gateway) for container image pulls (ghcr.io) and AWS API endpoints (SSM, CloudWatch, ECR)
• Public subnets must have an Internet Gateway route (only needed if alb_scheme = "internet-facing")
• Subnets should span at least 2 AZs for RDS Multi-AZ and ECS placement

alb_security_group_id = "sg-0abc1234def56789a"
ecs_security_group_id = "sg-0abc1234def56789b"

Git webhooks on a private install

alb_scheme             = "internal"
enable_webhook_ingress = true
webhook_providers      = ["github", "gitlab"]   # default ["github"]

• One route per provider, POST /api/v1/webhooks/<provider>/mcp/{sync_id}. Everything else, including other methods, other paths and .. segments, gets 404 from API Gateway and never reaches the VPC, so the UI, API and login stay private.
• A VPC link in the private subnets that connects API Gateway to the ALB’s listener, so there are no public subnets, internet gateway or second load balancer. With enable_tls, API Gateway connects over HTTPS and checks the ALB certificate against domain_name.
• A security group rule that lets the VPC link, and nothing else new, reach the ALB.
• Throttling (webhook_throttle_rate_limit, webhook_throttle_burst_limit), access logs in CloudWatch, and WEBHOOK_PUBLIC_URL on the API so the MCP Sync tab shows the public webhook URL.

• The ALB must be internal. API Gateway connects to the addresses the ALB’s DNS name resolves to, and an internet-facing ALB resolves to public addresses the VPC link cannot reach; Terraform stops with an error in that case. An internet-facing ALB that is open to the internet does not need this.
• There is no source IP allowlist. GitHub and GitLab do not sign their source addresses, and the providers’ ranges change, while every delivery is already authenticated. If you need one, put a REST API with a resource policy, or AWS WAF, in front instead.
• A VPC link with no traffic for 60 days goes inactive, and the next delivery fails while AWS reprovisions it, which takes a few minutes. GitHub does not retry failed deliveries automatically; use Redeliver on the hook, or run the sync by hand from the Sync tab.
• API Gateway throttling is approximate and meant as a backstop. Requests over the limit get 429 and show as failed in the provider’s delivery log.
• VPC links are not available in every Availability Zone (for example use1-az3). If terraform apply reports an unsupported zone, use private subnets in other zones.
• Turning it on adds WEBHOOK_PUBLIC_URL to the API task definition, which rolls the API tasks once.
• Cost: about $1 per million deliveries; the VPC link itself is free.

Required IAM permissions

Operating the install

Tail an ECS service

aws logs tail /aws/ecs/observal-prod/api --follow

terraform output log_group_names

Force a rolling deploy

aws ecs update-service \
  --cluster $(terraform output -raw ecs_cluster_name) \
  --service observal-prod-api \
  --force-new-deployment

Re-run migrations / seeds

$(terraform output -raw init_run_task_command)

Shell into the data host

$(terraform output -raw data_host_ssm_session_command)
# inside the session:
sudo journalctl -u observal-bootstrap -f                                 # cloud-init / first-boot logs
sudo docker compose -f /opt/observal/docker-compose.data.yml ps          # CH/Grafana/Prom status
sudo docker compose -f /opt/observal/docker-compose.data.yml logs -f clickhouse

Read a generated secret

aws ssm get-parameter --with-decryption \
  --name /observal-prod/SECRET_KEY \
  --query Parameter.Value --output text

Upgrade to a new Observal release

# terraform.tfvars
image_tag = "v1.5.0"

terraform apply

Resize the data volume

# terraform.tfvars
data_volume_size_gb = 250

terraform apply
# then SSM into the data host:
sudo growpart /dev/nvme1n1 1 || true
sudo resize2fs /dev/nvme1n1

Destroy

terraform destroy

Remote state

1. Create an S3 bucket (versioned, encrypted) and a DynamoDB table with hash key LockID.
2. Uncomment and fill the backend block in versions.tf:
   backend "s3" {
     bucket         = "your-tf-state-bucket"
     key            = "observal/prod/terraform.tfstate"
     region         = "us-east-1"
     dynamodb_table = "your-tf-lock-table"
     encrypt        = true
   }
   
3. Re-run terraform init and answer “yes” when prompted to migrate state.

Cost expectations

Production hardening checklist

• Switch to remote state (S3 + DynamoDB)
• Restrict alb_ingress_cidrs to known networks
• Enable AWS GuardDuty + Config in the account
• Add CloudWatch alarms on RDS CPUUtilization, FreeableMemory, ECS service CPU, ALB HTTPCode_Target_5XX_Count
• Attach AWS WAF to the ALB
• Set transit_encryption_enabled = true on the ElastiCache replication group and switch REDIS_URL to rediss://...
• Configure Observal SSO. See Authentication and SSO
• Test the backup and restore procedure end-to-end
• Replace the GitHub tarball download in user-data.sh.tftpl with an artifact URL you control

Troubleshooting

aws ecs describe-services \
  --cluster $(terraform output -raw ecs_cluster_name) \
  --services observal-prod-api \
  --query 'services[0].deployments'
aws logs tail /aws/ecs/observal-prod/api --follow

• RDS not yet reachable (transient on first apply; re-running fixes it)
• Migration error (look at the entrypoint output)

• For api: health check hits /readyz on port 8000. Tail the api log group for startup errors.
• For web: health check hits / on port 3000. Check the web log group.
• For Grafana: health check hits /api/health on port 3001. SSM into the data host and check docker compose ps.