sync schema with production as base schema by aagbsn · Pull Request #521 · ooni/devops

aagbsn · 2026-10-23T08:45:04Z

Sync scripts/cluster-migration/schema.sql with production's tables. The intent is to use it as a single source of truth for both CI and production deploys. This PR makes it an exact copy of production; the changes that make it reusable come in follow-ups (below).

How to review

Commit by commit. The first two only reformat (with clickhouse-format, the layout production's DDL uses) and sort by table: formatting the old and new files gives identical statements. Each later commit changes, adds or removes one table, and its message says what differs from main and why. Worth a careful look:

citizenlab: production's Keeper path has no /{shard}. A replica created from main's path wouldn't join production's table.
citizenlab_flip: removed; production doesn't have it.
event_detector_cusums: production's is a plain ReplacingMergeTree(ts), neither replicated nor partitioned.
fastpath: the #1279 and #1292 changes are in the CREATE; the ALTERs that added and then dropped four columns are gone.

Where the definitions come from

Dumped from data1, data2 and data3 on 2026-10-20, after asnmeta was made replicated again on data1 (OOPRIV-181). Loaded into a one-node oonidata_cluster on ClickHouse 24.8.6.70, the file creates all 21 tables and views as production reports them, Keeper paths included. The header says which replica's definition is used where they differ, and what's left out. Main's column comments are kept.

Nothing here runs against production: it's a change to the file only.

Follow-ups

Numbered migrations in scripts/cluster-migration/migrations/, applied once, instead of ALTERs appended to schema.sql, with a CI check that schema.sql equals the baseline plus every migration.
Unqualified table names, so clickhouse-client --database decides (e.g. for CI), and {database} in every Keeper path, so a scratch database can't join production's replicas. Tested on a two-node 24.8 cluster: the node that receives an ON CLUSTER query qualifies unqualified names with its own database before passing it on, and this schema made generic that way loads into ooni with exactly the definitions and Keeper paths it has here, and into a second database on the same cluster with no shared path.
An explicit target table for counters_asn_test_list instead of its .inner_id table.
The backend's test fixtures generated from schema.sql.

Run every statement through clickhouse-format (24.8), the layout SHOW CREATE and production's DDL use, so the per-table commits that follow show only real changes. Comments are kept. No statement changes: formatting the old and new files gives identical output.

Move each statement next to the others of its table, tables by name, ALTERs after their CREATE. Only the order changes.

Add the columns production has: top_{dns,tcp,tls}_rule_id, probe_id and analysis_rules_version, and its minmax indexes on measurement_start_time, probe_cc and probe_asn. Like every table from here on, it's now created as ooni.<table> with IF NOT EXISTS, as production reports it.

Replicated at /clickhouse/{cluster}/tables/ooni/asnmeta on every replica, as ooni/data#190 made it (and OOPRIV-181 restored on data1).

No change but the explicit index_granularity production reports.

Production's path has no /{shard} suffix. A replica created with the old path wouldn't join production's table.

Production doesn't have it. Only the legacy citizenlab updater used it, and ooni/data's updater (ooni/data#190) swaps citizenlab without it.

Production has none of the 17 *_obs_w_sum, *_w_sum, *_s_pos, *_s_neg, last_ts and current_mean columns, and no partitioning. It has current_state, and block_type in the CREATE instead of a later ALTER. float is now spelled Float32, the same type.

Production's table is a plain ReplacingMergeTree(ts), not replicated or partitioned, with *_current_state instead of *_obs_w_sum and *_w_sum. The *_last_change and *_last_ts columns are data1's, the only replica with rows.

No change but the explicit index_granularity production reports.

anomaly, confirmed and msm_failure are LowCardinality(String) (ooni/backend#1279), input has a bloom filter index (ooni/backend#1292), and is_verified is in the CREATE. The ALTERs that added and then dropped nym, zkp_request, age_range and msm_range are gone. data1 and data3's definition: data2 alone also has deduplicate_merge_projection_mode = 'rebuild'.

No change but the explicit index_granularity production reports.

An EmbeddedRocksDB table, so it isn't replicated: production has it on data1 and data2 only, and ON CLUSTER creates it, empty, on every node.

An EmbeddedRocksDB table, not replicated, like fingerprints_dns.

No change but IF NOT EXISTS.

A local MergeTree. data1's definition, the only replica with rows and with failed_fastpath.

Replicated at production's path, msmt_feedback_new.

Replicated at production's path, obs_web_repl.

A Join table, local to each node.

Its inner table keeps production's Keeper path, so a new replica joins the existing one. Production has the view on data1 and data3.

A header with the dump date, which replica's definition is used where they differ, and what's left out: the updaters' *_tmp tables, msmt_feedback_old, session_expunge (unused since ooni/backend#972) and the leftover inner table of the dropped counters_test_list view. Also CREATE DATABASE, which the ooni. names need on a new cluster.

aagbsn · 2026-10-23T09:15:16Z

+    `input` String,
+    `msmt_cnt` UInt64
+)
+ENGINE = ReplicatedSummingMergeTree('/clickhouse/{cluster}/tables/ooni/.inner_id.6a72ceec-7e0e-4b65-a3ab-da2809202114/{shard}', '{replica}')


This keeper path is needed for a rebuilt replica added to the cluster, but using an explicit target table name has other benefits than just being more readable:

Explicit target tables (… TO ooni.counters_asn_test_list_data) are still worth doing, for other reasons:

The definition reads plainly, with no generated name.

The view can be dropped and recreated, for example to change its query, without losing its rows.

The target can be altered or backfilled directly.

Backups and restores name the table directly instead of going through the view.

ClickHouse fills in {database} when it creates a table, so in production the paths are unchanged: /clickhouse/{cluster}/tables/ooni/<table>/{shard}. A second database on the same cluster, e.g. a scratch or CI one, now gets paths of its own instead of joining production's replicas. counters_asn_test_list's path keeps production's name for its hidden table, .inner_id.6a72ceec-…, with a comment saying why: a replica added later joins production's table whatever UUID its view gets. With {table} there, it would start a separate, empty path unless its CREATE copied the view's UUID.

Table names are unqualified, so clickhouse-client --database decides where the file loads, and CREATE DATABASE moves to the header's instructions: it has to exist on every node first, or the nodes without it fail with UNKNOWN_DATABASE while the others create the table. Tested on a two-node 24.8.6.70 cluster with Keeper. The node that receives an ON CLUSTER query qualifies its names with the client's database before passing it on, so every node uses that database, not its own default. Loaded into ooni, this file gives exactly the definitions and Keeper paths of the previous commit's file on both nodes; loaded again into ooni_test on the same cluster, it shares no path with ooni, and its view reads ooni_test's tables.

hellais · 2026-10-27T13:30:30Z

+    `notes` String
 )
-ORDER BY (ts, type, probe_cc, probe_asn, uid);
+ENGINE = EmbeddedRocksDB


We should be very careful about this, since clickhouse does not handle replication for it natively and it requires manually synching the data between all replicates. Alternatively since we now have a cluster setup we could change this to a different engine type.

yes - we have a backup job that does serialize to/from this type but I want to move to replicated types instead.

While a materialized view exists without its target table, every insert into the view's source table fails: for counters_asn_test_list, fastpath's inserts. Tested on 24.8. It matters once #521's follow-up gives counters_asn_test_list an explicit target table.

aagbsn added 25 commits October 23, 2026 08:39

Sort schema.sql by table name

9b0853c

Move each statement next to the others of its table, tables by name, ALTERs after their CREATE. Only the order changes.

schema.sql: add asnmeta

b6526fe

Replicated at /clickhouse/{cluster}/tables/ooni/asnmeta on every replica, as ooni/data#190 made it (and OOPRIV-181 restored on data1).

schema.sql: changepoint_label as in production

8e8197a

No change but the explicit index_granularity production reports.

schema.sql: citizenlab's Keeper path as in production

8e89a4e

Production's path has no /{shard} suffix. A replica created with the old path wouldn't join production's table.

schema.sql: drop citizenlab_flip

6137465

Production doesn't have it. Only the legacy citizenlab updater used it, and ooni/data's updater (ooni/data#190) swaps citizenlab without it.

schema.sql: event_detector_cusums as in production

4d1f616

Production's table is a plain ReplacingMergeTree(ts), not replicated or partitioned, with *_current_state instead of *_obs_w_sum and *_w_sum. The *_last_change and *_last_ts columns are data1's, the only replica with rows.

schema.sql: event_detector_v2_changepoints as in production

3ccfcb6

No change but the explicit index_granularity production reports.

schema.sql: faulty_measurements as in production

dc8fb71

No change but the explicit index_granularity production reports.

schema.sql: add fingerprints_dns

5552cc1

An EmbeddedRocksDB table, so it isn't replicated: production has it on data1 and data2 only, and ON CLUSTER creates it, empty, on every node.

schema.sql: add fingerprints_http

0768d6b

An EmbeddedRocksDB table, not replicated, like fingerprints_dns.

schema.sql: jsonl as in production

1048e6c

No change but IF NOT EXISTS.

schema.sql: add labeling_frames

534449c

A local MergeTree. data1's definition, the only replica with rows and with failed_fastpath.

schema.sql: add msmt_feedback

473d7fb

Replicated at production's path, msmt_feedback_new.

schema.sql: add obs_http_middlebox

49360bb

schema.sql: add obs_openvpn

b435644

schema.sql: add obs_web

014b4e2

Replicated at production's path, obs_web_repl.

schema.sql: add obs_web_ctrl

46a1f80

schema.sql: add test_groups

53a6e66

A Join table, local to each node.

schema.sql: add url_priorities

edfd3ac

schema.sql: add the counters_asn_test_list view

b6c37ad

Its inner table keeps production's Keeper path, so a new replica joins the existing one. Production has the view on data1 and data3.

aagbsn commented Oct 23, 2026

View reviewed changes

aagbsn changed the title ~~Schema sql current state~~ Oct 23, 2026

aagbsn marked this pull request as draft October 23, 2026 09:21

aagbsn added 2 commits October 23, 2026 09:39

hellais reviewed Oct 27, 2026

View reviewed changes

aagbsn mentioned this pull request Oct 28, 2026

Clickhouse restore #527

Draft

Navigation Menu

Repository navigation

sync schema with production as base schema - #521

sync schema with production as base schema#521
aagbsn wants to merge 27 commits into
mainfrom
schema_sql_current_state

aagbsn commented Oct 23, 2026 •

edited

Loading

aagbsn Oct 23, 2026

hellais Oct 27, 2026

aagbsn Nov 2, 2026

Uh oh!

Labels

2 participants

Conversation

aagbsn commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to review

Where the definitions come from

Follow-ups

aagbsn Oct 23, 2026

Choose a reason for hiding this comment

hellais Oct 27, 2026

Choose a reason for hiding this comment

aagbsn Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

Labels

2 participants

aagbsn commented Oct 23, 2026 •

edited

Loading