Skip to content

measurements: bound single-measurement lookups by start time - #1311

Open
aagbsn wants to merge 5 commits into
masterfrom
measurement_lookup_time_window
Open

aagbsn wants to merge 5 commits into
masterfrom
measurement_lookup_time_window

Conversation

@aagbsn

@aagbsn aagbsn commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Looking up one measurement by measurement_uid scans most of fastpath. Its primary key starts with measurement_start_time, and the only index on measurement_uid is a minmax skip index. On data2 it reads the most bytes of any query, three times the next: 31,994 /v1/measurement_meta and /v1/raw_measurement lookups in 19.8 h, reading 1.1 GB each (p50 237 ms), about 36 TB in total (system.query_log, 2026-10-20 12:04 to 2026-10-21 07:54 UTC).

A measurement's start time is close to times we already know. The measurement_uid starts with the time it was collected, and the report_id with the time its report was opened. So the lookups search widening windows of measurement_start_time around that time, which the primary key prunes, and finally all of fastpath. The result doesn't depend on the windows.

Changes, in oonimeasurements only:

  • _query_one_in_windows runs a query with each window in turn, then unbounded, and returns the first row found.
  • Lookup by uid (_get_measurement_meta_by_uid): ±15 min, then -1 day/+1 h. In production, measurements are collected a median 3 s after they start (p99 216 s, p99.9 about 4 h), and 3% start slightly after (fast probe clocks).
  • Lookup by report_id and input (_get_measurement_meta_clickhouse): -15 min/+2 h, then -2 days/+1 day. In production 59% of measurements start before their report_id time, mostly by seconds (p1 -107 s), and the rest after (p99 +1.7 h, p99.9 +14 h).
  • measurement_uid_to_s3path_linenum finds the measurement's (report_id, input) in fastpath through the uid windows, then reads jsonl by its primary key. Before, it bounded the fastpath subquery inside the jsonl query. A measurement that jsonl doesn't have yet, like one newer than the last can upload, would then miss every window and still scan all of fastpath.

jsonl is still looked up by (report_id, input), not measurement_uid: production jsonl rows written by the API uploader between #192 and #273 (around 2021) have an empty measurement_uid.

Benchmark on production's table definitions, with 120M production-shaped rows over 60 days (ClickHouse 25.2, make bench):

case MB read before MB read after
measurement_meta by uid 400.0 9.3
measurement_meta by report_id 325.4 17.4
the other 53 cases same same

All responses are identical. Granules read per lookup, for 100 random measurements of the last 7 days (tests/bench/lookup_pruning.py), median: by uid 728 -> 6, by report_id 608 -> 24.

Tests:

  • Measurements found by the first window, by the second, and only by the unbounded query, for each lookup.
  • A uid without a timestamp prefix.
  • Which queries each lookup runs when the first window finds the measurement, including the jsonl miss. This test fails on the previous version.

The lookup tests' rows are inserted through insert_fastpath and a new insert_jsonl fixture, and are deleted on teardown.

Replaces #1276, which was stacked on the benchmark harness (since merged as #1274) and on #1285 (closed).

most measurements are uploaded close to their start time.
this uses widening lookup windows to avoid scanning many granules,
because the skip index is already at minimum and shrinking granules
makes the primary index much larger plus requires a rebuild of the
table.

another option is to use a bloom_filter index or a projection, but these
adds overhead on every insert
Benchmark on production's table definitions, ClickHouse 25.2, 120M
production shaped rows over 60 days, two rounds of three repetitions,
with the two previous commits; all responses identical:

    measurement_meta.uid        400.0 -> 9.3 MB read (43x)
    measurement_meta.report_id  325.4 -> 17.4 MB read (19x), 1.5x faster
    other 53 cases              same bytes read

Granules read per lookup, 100 random measurements of the last 7 days
(tests/bench/lookup_pruning.py), median: by uid 728 -> 6, by report_id
608 -> 24.
The lookup tests' session fixture left web_connectivity rows in fastpath,
and the s3path tests left their rows in jsonl. Insert them through
insert_fastpath, under their own test_name, and a new insert_jsonl
fixture, which delete them on teardown. Name the rows by the window
that finds them.
…jsonl

measurement_uid_to_s3path_linenum bounded the fastpath subquery of its
jsonl query, so a measurement fastpath has but jsonl doesn't yet, as
those newer than the last can upload, missed every window and fell
through to the unbounded scan of fastpath. Look up its report_id and
input in fastpath through the windows, then read jsonl by its primary
key.

Test which queries each lookup runs when the first window finds the
measurement.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant