Repository navigation
Conversation
- Install docker-ce from Docker's official apt repo (distro-detected via ansible_distribution_release, not hardcoded), replacing Debian's docker.io/podman with containerd's nftables firewall backend enabled. Verify the downloaded apt signing key's fingerprint before trusting it. - Rewrite the deploy stage as a single dependency-free deploy.py (stdlib only, shells out to ssh/scp/aws/docker) instead of inline buildspec bash. - Render one docker-compose.yaml per slot instead of a Quadlet unit; slots are supervised by docker's own restart policy, no systemd unit needed. - Move service secrets from a shared env file to Compose's native file-based secrets, so each is mounted read-only at /run/secrets/<name> instead of ending up in the container's environment. - Stage files scp'd to target hosts in a private, deploy-user-owned directory (chmod 600 + scp -p) instead of world-readable /tmp, closing the window where they'd be briefly readable by other local users.
main replaced the shared clickhouse_write_url, clickhouse_readonly_url and clickhouse_readonly_test_url parameters with one per service (b7baff6). The blue/green service secrets are built from the same parameters as each service's ECS task_secrets, so they now point at the per-service ones again: ooniprobe, oonirun and oonimeasurements in dev, oonirun in prod. dev oonifindings no longer gets a CLICKHOUSE_URL, like its ECS task. Without this, terraform init fails on references to undeclared data sources after merging main.
Review: "We should add a log.info on this" (on run()). Every ssh, scp and aws command now goes through the logger with a timestamp, so the CodeBuild log shows what ran against which host and when, not just the start and end of each host's deploy. Secret values are only ever written to files, never put on a command line, so they don't reach the log.
Review: "We should make sure 20s is enough". It wasn't comparable to what the same images get on ECS: the target groups (ooniapi_service) check /health every 30 s with a 5 s timeout and want 2 passes in a row, so a new task has about a minute to come up. The blue/green check gave 20 s, a single pass sufficed, and curl had no timeout of its own. Now each check has curl --max-time 5, the slot must pass twice in a row, and it has health_check_timeout seconds (default 120, per service) to do so. The time it took is logged, so the deadline can be tuned per service from real deploys; at 2 hosts per deploy it stays well within the deploy job's 20 minute build timeout.
Review: "what happens if nginx -t fails". The new upstream conf was moved into conf.d before nginx -t, and left there when it failed. nginx kept serving its old config, but the next reload on that host (another service's deploy, a certificate renewal) would fail, and a restart would take every service on the host down. Now a failing nginx -t puts back the upstream of the slot that is still serving and checks it again, then aborts the deploy without reloading nginx or moving the active_slot marker. If nginx -t fails even then, the config is broken independently of this deploy, and the error says so. The restore uses the same mv the deploy user's sudoers already allows.
Review: "We should make sure there is some kind of timeout on connections while draining". After a flip, nginx workers from before the reload keep serving their in-flight requests to the old slot, with nothing bounding them but proxy_read_timeout (900 s), and long lived HTTP/2 connections can keep them longer. On ECS the ALB bounds the same thing with its deregistration delay, the AWS default of 300 s here. - ooniapi_gateway sets worker_shutdown_timeout to ooniapi_gateway_drain_timeout (300 s), through modules-enabled/, the only place included in nginx's main context. - The next deploy of a service replaces the slot that is draining, so deploy.py now waits until drain_timeout (300 s, per service) has passed since the previous flip, from the active_slot marker's mtime, before restarting it. A first deploy has no marker and does not wait.
The routes of the API are going to be served by two frontends, the ALB for the services on ECS and the nginx gateway on the Hetzner hosts for those deployed blue/green, and their copies have already drifted: the gateway's hand written locations predate main's X-Protocol-Version rules. routes.yaml becomes the one place they are declared; this change has Terraform read it, the gateway follows. The 18 hand written listener rules become one aws_lb_listener_rule with for_each over routes.yaml, with moved blocks from their old addresses. Planned with fake ARNs (no state), the old and the new code produce the same 18 rules: same priorities, targets, conditions and tags. Applying should only move them in the state. The "hotfix" that made the oonimeasurements and testlists rules optional (count = arn != null ? 1 : 0) is removed rather than carried over. It had stopped working anyway: oonimeasurements_rule_3 lacked the count, so the frontend couldn't be planned without oonimeasurements. Both environments pass every target group, so those two inputs are now required and nullable = false. A missing service fails the plan with "Required variable not set" instead of silently losing its routes. The planned rules are unchanged.
The gateway's locations were a hand written copy of the ALB rules, and had already fallen behind them (main's X-Protocol-Version rules). They are now generated from tf/modules/ooniapi_frontend/routes.yaml, which the ALB rules are generated from too, as are the per service direct-host vhosts. nginx can't follow ALB priorities: it picks an exact match, then the longest prefix. filter_plugins/ooniapi_routes.py turns each ALB pattern into an exact or ^~ prefix location and lists the routes nginx couldn't serve the same way (a wildcard other than a trailing *, the same path twice, a prefix of one service covering a path of another); the role fails on any. A plugin rather than inline Jinja, so the result doesn't depend on how a given ansible-core converts templated strings. Routes marked legacy (ooniprobe_legacy, being retired) are left to the ALB: the gateway doesn't run it, so those paths go to ooniprobe. Rendered with ansible-core 2.21 for all six services plus testlists proxied cross-cloud, the generated config has the same 46 locations, with the same match types and upstreams, as the hand written one.
nginx only allows letters, digits and underscores in a syslog tag, so
nginx -t rejected the gateway vhost ("syslog \"tag\" only allows
alphanumeric characters and underscore", nginx 1.27) and it could never
have been installed. The tag is now ooniapi_gateway.
Nothing caches oonimeasurements or oonirun today: on api.ooni.org every repeated request reaches uvicorn (only the reverseproxy's /api/_/ paths show x-cache-status: HIT), and for oonimeasurements that means ClickHouse, saturated on data2. The gateway now caches GET/HEAD responses of the routes routes.yaml marks cache (oonimeasurements, oonirun), as long as their Cache-Control allows, or 10 s without one, like the legacy ooni-api vhost's apicache. Requests with an Authorization header never use the cache (oonirun links can differ per user), and nginx doesn't store no-cache, private or max-age=0 responses, or any setting a cookie. Two zones, since nginx keeps an entry until it goes unused for inactive, expired or not: lists and aggregations valid for minutes would otherwise push out measurement bodies valid for a day. Sized per host as if it got all the traffic, from data2's query_log and responses of api.ooni.org: - long (raw_measurement, measurement_meta; valid 1 d): at most ~50K distinct bodies a day of ~76 KB (max seen ~160 KB), ~3.8 GB a day: max_size 8g, keys_zone 16m (~130K keys), inactive 1d. - short (everything else cached; valid 10 s to 1 h): ~7.5K requests an hour of up to ~60 KB, ~450 MB an hour if all were distinct: max_size 2g, keys_zone 8m, inactive 1h. The access log now records $upstream_cache_status, to tune these from real hit rates. Cache directives and X-Cache-Status are set per server, since an add_header in a location would drop the server's HSTS headers. Checked with nginx 1.27 and stub upstreams: lists, bodies (long zone) and header-less aggregations miss then hit; Authorization bypasses and isn't stored; no-cache isn't stored; ooniprobe and the default route aren't cached; every response keeps HSTS.
A deploy can change what a service returns, but the gateway would keep serving responses cached from the old slot, for up to a day for measurement bodies. nginx without the commercial purge API can't drop entries, so each service's cache key now starts with a version: the image tag, which deploy.py writes into <service>-upstream.conf as $ooniapi_cache_version_<service>, the same file and the same reload that put the new slot in rotation. From then on the old entries are never looked up and age out of the zones; other services' caches are left alone. (The slot letter wouldn't do: slots alternate, so it would bring back responses from two deploys ago.) When nginx -t rejects the new upstream conf, deploy.py now puts the previous file back as it was, cache version included, rather than rendering it again. The image tag is checked to be a plain tag, since it ends up in an nginx string. The placeholder upstream conf the role seeds before a first deploy defines an empty version. Checked with nginx 1.27: after a deploy of oonimeasurements its cached list missed (new upstream response) then hit again, while oonirun's cached response kept hitting; redeploying the previous tag found that tag's entries again, as long as they were still valid.
A service had to choose between ECS and the Hetzner hosts. Switching one to "blue_green" removes the ECS deploy action but leaves the ECS service running, so the ALB keeps sending api.ooni.org and <service>.prod.ooni.io traffic to tasks that no longer get new versions, until DNS moves to the hosts. "both" deploys the same build artifact to ECS and then, in the same Deploy stage, runs the blue/green deploy to the hosts. The ECS deploy runs first, so the hosts never get a version ECS rejected. Either failing fails the pipeline. The migration can then keep the single DNS cutover of the original design while every service runs in "both": the hosts are proven before the move, and DNS can go back to the ALB, which still serves the current version, if they aren't. "ecs" and "blue_green" plan exactly as before. Planned with fake inputs, one service per mode, and the account id lookup replaced by a constant: the 6 and 13 resources are identical. A "both" service gets the 13 blue/green resources and a Deploy stage with ECS "Deploy" (run order 1) then CodeBuild "DeployBlueGreen" (run order 2). Both actions need their own name and namespace. The pipeline role already allows ecs:* and codebuild:StartBuild on any resource. The per-service comments in dev and prod now mention "both", and no longer say the blue/green deploy uses systemd (it uses Docker Compose since e11b071).
aagbsn
marked this pull request as draft
October 4, 2026 04:56
Terraform Run Output 🤖Format and Style 🖌
|
| Pusher | @aagbsn |
| Action | pull_request |
| Environment | dev |
| Workflow | .github/workflows/check_terraform.yml |
| Last updated | Sun, 04 Oct 2026 06:41:36 GMT |
Applying prod as it was would create 7 Secrets Manager secrets: a JSON copy of each service's credentials (Postgres URL, JWT key, ClickHouse URL, ooniauth's AWS keys) and the deploy SSH key container. Every prod service is still deploy_mode = "ecs", so nothing would read them. They are commented out in place, with each deployer's pre-wired env_vars/secrets block, so moving a prod service to "blue_green" or "both" starts by uncommenting both. Against main, tf/environments/prod now differs only by comments and the explicit deploy_mode = "ecs" (the default), so the prod plan of this branch is the ooniapi_frontend listener rules moving to their routes.yaml addresses: planned against a state built from main's module with prod's inputs, 18 moved, 0 to add, change or destroy. terraform validate and fmt pass.
Dev needs one Hetzner host: an outage while it reboots costs nothing, and backend-hel is free for it. With both hosts required, a dev service would have to name backend-hel twice, and deploy.py would then deploy to it twice, the second time waiting out the drain timeout (300 s) for the slot it had just taken out of rotation. Without deploy_host_secondary the deploy job gets no DEPLOY_HOST_SECONDARY and deploy.py deploys to the primary alone. Planned with the deployer test module in each mode, a service with both hosts plans the same resources as before (only the deploy.py object changes); without the secondary, only that environment variable is left out.
backend-hel has been re-imaged and is free; one host is enough for dev. Until now nothing said where a dev service would go: the pre-wired deployers had no host, ports, network or deploy bucket, and the gateway playbook targeted backend-fsn, which still serves the legacy API, as well. - tf/environments/dev: each deployer gets backend-hel as its only host, its slot ports, the ooniapi docker network and container port 80, and a deploy bucket for the compose files, upstream confs and deploy.py. The services stay deploy_mode = "ecs", so the only resource this adds is the bucket. - host_vars/backend-hel.ooni.org/ooniapi_gateway.yml: the gateway answers for api.dev.ooni.io and <service>.dev.ooni.io, with the same ports. testlists stays on its EC2 instance, which serves only plain HTTP, so the gateway proxies it over HTTPS through the ALB at testlists.dev.ooni.io; that name stays on the ALB at the DNS cutover. The dev names' certificate can only be issued after the cutover; the file says how. - deploy-ooniapi-gateway.yml targets backend-hel only, and installs the nginx and dehydrated roles the gateway role expects first. Rendered with these settings, the gateway vhost passes nginx -t (nginx 1.27) with all routes.yaml locations, the five direct-host vhosts and the testlists passthrough. terraform validate and fmt pass for dev.
The role gave the deploy user its staging directory and sudoers rule but never created it, so on a fresh host (backend-hel was just re-imaged) its first task owning a path by that user fails, and the deploy job has no account to log in to. The role now creates the user, without groups, and authorizes only ooniapi_gateway_deploy_ssh_public_key, the public half of the key in the environment's oonidevops/ooniapi/deploy_ssh_key secret. It stops with instructions while that is unset; for backend-hel it is left empty until the key pair exists. Run in a Debian bookworm container with the pinned ansible 9.3.0: creates the user and key, a second run changes nothing, and an empty key fails with the message.
oonimeasurements keeps its rate limit counters in valkey, on ECS the ElastiCache Serverless cache, which only answers inside the VPC: from a Hetzner host the service could not reach it. The gateway role now runs a valkey container named valkey on the shared docker network, without a published port, in memory only (save and appendonly off, 256 MB, allkeys-lru): losing the counters on a restart only resets the limits. Dev's oonimeasurements blue/green env uses valkey://valkey:6379; ECS keeps ElastiCache. Each host keeps its own counters, so with two hosts behind DNS a client's limit is per host. Checked with oonimeasurements' limits storage (RedisStorage with async+valkey://) against valkey/valkey:8.1-alpine started with these arguments: a 3/minute limit allows 3 hits and refuses the 4th and 5th.
The dev services on backend-hel reach the same backends as on ECS, but from outside the VPC: - fastpath: the host firewall the fastpath playbook installs already accepts 8472 (measurement submission) and 8475 (measurement spool) from anywhere; it is the instance's security group that only lets in the VPC. ooni_fastpath gains extra_ingress_cidrs (default none, so prod's three instances keep their rules), and dev passes backend-hel. The blue/green env of ooniprobe (FASTPATH_URLS) and oonimeasurements (OTHER_COLLECTORS) uses fastpath.dev.ooni.io instead of the instance's private IP, which only resolves to something reachable inside AWS. - Postgres (publicly accessible, open to 10.0.0.0/8 only) and the ClickHouse proxy (9000-9002, the VPC and listed instances) let backend-hel in too. backend-hel's addresses come from DNS, the way monitoring.ooni.org's do for the ClickHouse proxy. The ClickHouse hosts' own nftables allow list already has backend-hel. Measurements and the spool travel plain HTTP between Hetzner and AWS, as they do from probes. terraform validate and fmt pass for dev and prod.
oonimeasurements' blue/green env had valkey://valkey:6379 inline, a literal that silently depended on the container name the gateway role picks. It is now ooniapi_deploy_valkey_url, next to the docker network and slot ports in dev's deploy locals, which all have to match the host, and the role's container name is ooniapi_gateway_valkey_name, each pointing at the other.
Neither service uses Postgres: ooniauth sends login emails through SES and signs JWTs, oonimeasurements reads ClickHouse only. Their blue/green service secrets still carried POSTGRESQL_URL, copied from their ECS task_secrets, and the deploy job would write it to disk on the host. Removed from dev's service secrets and from prod's commented-out ones. prune_service_credentials removes the same variable from their ECS tasks. terraform validate passes for dev and prod.
The services' CORS middleware (Starlette) answers a request from an allowed origin with Access-Control-Allow-Origin set to that origin, and a request without an Origin, or from a disallowed one, with no CORS headers and no Vary: Origin. The cache key left Origin out, so whichever request filled an entry decided what every origin got: one curl request without Origin was enough to make browsers on the Explorer fail the response's CORS check, for up to a day in the long zone. The key now includes $http_origin, so each origin gets the response the service gave it. A client sending random origins only misses the cache, as random query parameters already would. Reproduced with the gateway rendered for backend-hel in nginx 1.27, in front of a FastAPI app with oonimeasurements' CORS settings, on /api/v1/aggregation: - before: a request without Origin, then one from explorer.ooni.org: HIT without Access-Control-Allow-Origin - after: the explorer.ooni.org request is a MISS answered with its Access-Control-Allow-Origin, and a HIT with it when repeated; an unknown origin gets no CORS header; test-lists.ooni.org gets its own
…easurements The gateway appends the address it received a request from to X-Forwarded-For, as the ALB does in append mode, so services behind it read the client's address from the last entry (ooni/backend client_ip_from_proxy). backend-fsn forwards ooniprobe and measurement API requests to ooniprobe.prod.ooni.io and oonimeasurements.prod.ooni.io; once those names point at the gateway, its address is the last entry for those requests, as it is behind the ALB today. The commented-out env for both services now carries TRUSTED_PROXIES with backend-fsn's address, matching their ECS tasks (ooniprobe_trusted_proxies), so it comes back when a prod service moves to "blue_green" or "both". Dev needs none: nothing forwards to the dev gateway.
…e file Each slot becomes /etc/containers/systemd/<service>-<slot>.container, run by rootful Podman with UserNS=auto. Secrets become Podman secrets passed to the container as environment variables: the services' settings only read environment variables, so the compose file's /run/secrets files were never read.
Secrets go in a root-owned, mode 600 env file per slot, which the unit loads, instead of Podman secrets: creating those through sudo would need a wildcard name, and sudoers' "*" also matches spaces, so a caller could add --driver shell and run a command as root. The deploy user's sudoers rule now lists each command it runs, per service and slot, with no wildcards.
Rootful Podman with Quadlet, netavark on nftables, the services' network as a Quadlet .network unit that the slots' units depend on, and Valkey as a Quadlet unit. This also drops the nftables setting written to containerd's config, which Docker never read. Docker, if an earlier run installed it, is removed.
Each service gets its own IPv4 network, ooniapi-<service>, with a /28 set in host_vars; Valkey joins only the networks of the services that use it. Slices cap CPU and memory: dev's slots and Valkey run in ooniapi-dev.slice (4 CPUs, 4 GB, CPU weight 20). reverseproxy leaves backend-hel's list, since it doesn't move to the hosts.
The gateway doesn't fall back to the legacy API through reverseproxy, as the ALB does: what's missing is ported instead.
…ranges Found running the role on a fresh trixie host: Debian's Podman packages create neither /etc/containers/containers.conf.d nor the subuid and subgid ranges for the user "containers" that UserNS=auto takes containers' IDs from, so the netavark setting couldn't be written and no container started.
flush ruleset also deleted other tables, so a reload cut off containers by removing netavark's rules (and Docker's NAT on hosts that run it).
The input chain drops by default, and containers reach aardvark-dns on their bridge's address, which is host input.
A cadvisor role (the release binary; Debian doesn't package it) reports each container's CPU, memory, network and OOM kills. Both exporters sit behind nginx on 9001, as on the test helpers, and Prometheus scrapes them.
deploy_mode "both": ECS deploys first and keeps serving through the ALB; then the same build goes to backend-hel's blue/green slots. DNS doesn't change. oonifindings goes first because it only needs Postgres.
It sits next to the private key's secret in the dev account, as /oonidevops/ooniapi/deploy_ssh_public_key, so rotating the pair needs no commit.
As with oonifindings, deploy_mode "both" keeps ECS and the ALB serving. ooniprobe waits: it reads S3 with the ECS task role's credentials, which the hosts don't have.
On ECS ooniprobe reaches its buckets with the task role; on the hosts it gets an IAM user with the same access (read the config and anoncred manifest buckets, write failed reports), as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in its host service secret. Its /health reads the anoncred manifest, so without them the slot would never go live.
|
A review by Opus 5.5 as requested by @aagbsn Review of ooni/devops PR 505 (blue/green deploys to Hetzner)The ALB refactor is correct, but the blue/green path has three problems I'd fix before relying on it: a CI build can steal a key that gives root on backend-hel, What's solid: High
Medium
Low and nits
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merged
main, then:Review comments addressed (one commit each)
deploy.pylogs every ssh/scp/aws command it runs (secrets are only written to files, never logged).curl --max-time 5, two passes in a row,health_check_timeout(default 120 s, per service) instead of a single 20 s check.
nginx -tfails, the live slot's upstream is restored and the deploy aborts without reloading nginx, insteadof leaving a broken conf for the next reload.
worker_shutdown_timeout300 s (like the ALB's deregistration delay), and the next deploywaits for the old slot to finish draining.
Fixes needed after merging
mainmainintroduced (otherwiseterraform initfails).ooniapi_gateway. nginx rejected the old tag, so the vhost could never have beeninstalled.
One source of truth for routes
tf/modules/ooniapi_frontend/routes.yamldeclares every API route.for_each, withmovedblocks. Planned against the old code:the same 18 rules, so applying only moves them in state.
route nginx can't serve the way the ALB does. This also picks up
main's X-Protocol-Version rules, which thehand-written gateway was missing.
Caching in the gateway
routes.yamlmarks for caching (oonimeasurements, oonirun), honoringCache-Control, 10 s when there is none. Skipped for requests with
Authorizationand forno-cache/private/Set-Cookie responses. Two zones, for short-lived and long-lived responses.
the old entries for that service only.
New
deploy_mode = "both"failure in either fails the pipeline.
both, DNS moves once, and the ALB still serves thecurrent version if DNS has to move back.
ecsandblue_greenmodes plan exactly as before.