[hellais]

Summary of changes:

• Parallelise on file entry instead of bucket_date. This leads to significant performance boost for processing measurements in a day
• Record inside of oonidata_processing_logs metrics about how long it took process a certain daily batch
• Change schema of HIRL to include sent and received
• Various dask related performance tuning
• Additional metric collection inside of statsd

Prior to this refactor we were achieving a rate of 5000 measurements per second, but were only reaching this speed when processing multiple bucket_dates at the same time.

This was happening because the parallelisation was happening on a per day bucket basic. When processing a single day worth of data (which is what happens in the daily batch), we were only reaching a speed of about 500 measurements per second.

As part of this PR I have added support for parallelising on a per archived file basis, which allows us to reach a performance of 7-8k measurements per second on a per day basis (which is actually faster than what we were doing before). The splitting into batches happens based on the size of the files so we should be getting a pretty consistent performance irrespective of the size of the daily batch.

This is the first step towards improving the performance of reprocessing of legacy buckets, where each daily bucket is too small to benefit from just splitting based on day (we ideally want to process batches of days together).

I also implemented as part of this same PR some changes to the schema of HIRL to make it more useful for analysis.

[hellais]

I came to the bottom of why the processing time for measurements in 2021 buckets was so slow. The root cause of it is ooni/backend#763.

In the interim, the refactoring in this branch does achieve marginally better performance, yet it's currently capped by the network (I am fully saturating the 1 GBiT link), see:

There isn't much more I can do except get a machine with a faster network or reprocess the postcans to compress them from a place that is closer to s3.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.94416% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.93%. Comparing base (410b5ae) to head (fe698ec).
⚠️ Report is 213 commits behind head on main.

Files with missing lines	Patch %	Lines
oonidata/workers/observations.py	48.88%	46 Missing ⚠️
oonidata/datautils.py	53.12%	15 Missing ⚠️
oonidata/dataclient.py	83.67%	8 Missing ⚠️
...a/transforms/nettests/http_invalid_request_line.py	0.00%	3 Missing ⚠️
oonidata/db/create_tables.py	75.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (62.94%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project status has failed because the head coverage (73.93%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #37      +/-   ##
==========================================
- Coverage   74.46%   73.93%   -0.54%     
==========================================
  Files          70       70              
  Lines        5883     5993     +110     
==========================================
+ Hits         4381     4431      +50     
- Misses       1502     1562      +60     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.