Repository navigation
Conversation
most measurements are uploaded close to their start time. this uses widening lookup windows to avoid scanning many granules, because the skip index is already at minimum and shrinking granules makes the primary index much larger plus requires a rebuild of the table. another option is to use a bloom_filter index or a projection, but these adds overhead on every insert
Benchmark on production's table definitions, ClickHouse 25.2, 120M
production shaped rows over 60 days, two rounds of three repetitions,
with the two previous commits; all responses identical:
measurement_meta.uid 400.0 -> 9.3 MB read (43x)
measurement_meta.report_id 325.4 -> 17.4 MB read (19x), 1.5x faster
other 53 cases same bytes read
Granules read per lookup, 100 random measurements of the last 7 days
(tests/bench/lookup_pruning.py), median: by uid 728 -> 6, by report_id
608 -> 24.
The lookup tests' session fixture left web_connectivity rows in fastpath, and the s3path tests left their rows in jsonl. Insert them through insert_fastpath, under their own test_name, and a new insert_jsonl fixture, which delete them on teardown. Name the rows by the window that finds them.
…jsonl measurement_uid_to_s3path_linenum bounded the fastpath subquery of its jsonl query, so a measurement fastpath has but jsonl doesn't yet, as those newer than the last can upload, missed every window and fell through to the unbounded scan of fastpath. Look up its report_id and input in fastpath through the windows, then read jsonl by its primary key. Test which queries each lookup runs when the first window finds the measurement.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Looking up one measurement by
measurement_uidscans most of fastpath. Its primary key starts withmeasurement_start_time, and the only index onmeasurement_uidis a minmax skip index. On data2 it reads the most bytes of any query, three times the next: 31,994/v1/measurement_metaand/v1/raw_measurementlookups in 19.8 h, reading 1.1 GB each (p50 237 ms), about 36 TB in total (system.query_log, 2026-10-20 12:04 to 2026-10-21 07:54 UTC).A measurement's start time is close to times we already know. The
measurement_uidstarts with the time it was collected, and thereport_idwith the time its report was opened. So the lookups search widening windows ofmeasurement_start_timearound that time, which the primary key prunes, and finally all of fastpath. The result doesn't depend on the windows.Changes, in
oonimeasurementsonly:_query_one_in_windowsruns a query with each window in turn, then unbounded, and returns the first row found._get_measurement_meta_by_uid): ±15 min, then -1 day/+1 h. In production, measurements are collected a median 3 s after they start (p99 216 s, p99.9 about 4 h), and 3% start slightly after (fast probe clocks)._get_measurement_meta_clickhouse): -15 min/+2 h, then -2 days/+1 day. In production 59% of measurements start before their report_id time, mostly by seconds (p1 -107 s), and the rest after (p99 +1.7 h, p99.9 +14 h).measurement_uid_to_s3path_linenumfinds the measurement's(report_id, input)in fastpath through the uid windows, then reads jsonl by its primary key. Before, it bounded the fastpath subquery inside the jsonl query. A measurement that jsonl doesn't have yet, like one newer than the last can upload, would then miss every window and still scan all of fastpath.jsonl is still looked up by
(report_id, input), notmeasurement_uid: production jsonl rows written by the API uploader between #192 and #273 (around 2021) have an emptymeasurement_uid.Benchmark on production's table definitions, with 120M production-shaped rows over 60 days (ClickHouse 25.2,
make bench):All responses are identical. Granules read per lookup, for 100 random measurements of the last 7 days (
tests/bench/lookup_pruning.py), median: by uid 728 -> 6, by report_id 608 -> 24.Tests:
The lookup tests' rows are inserted through
insert_fastpathand a newinsert_jsonlfixture, and are deleted on teardown.Replaces #1276, which was stacked on the benchmark harness (since merged as #1274) and on #1285 (closed).