Skip to content

Performance and API improvements - #6

Merged
hellais merged 65 commits into
mainfrom
mashumaro
Oct 18, 2022
Merged

Performance and API improvements#6
hellais merged 65 commits into
mainfrom
mashumaro

Conversation

@hellais

@hellais hellais commented Sep 21, 2022

Copy link
Copy Markdown
Member
  • Switch to using mashumaro and orjson for parsing raw measurements and validating/generating the dataclasses, which gives a ~30x performance boost (see: https://github.com/hellais/oonidata-bench)
  • Consume the measurements from the cans instead of the public JSONL files so as to preserve a valid measurement_uid and avoid producing duplicate measurements (Duplicate data present in some s3 buckets backend#613)
  • Make the API for iterating over measurements better
    • Add more precise progress bar
    • Where possible, yield the measurements while streaming instead of having to decompress and keep the whole can in memory
    • Make the API interface a bit more ergonomic to make it faster to use in live environments like jupyter
    • Native support for parallel processing of measurements
  • Several bug fixes to the parsers and dataformats by having more strict checks on the consistency of data
  • Add support for extracting more metadata related to resolver information
  • Nicer API for generating observations directly from a parsed base measurement
hellais and others added 30 commits August 9, 2022 15:06
@hellais
Fix tests for netinfodb
b568935
@hellais
Update create tables
e66924a
@hellais
Improvements to the oonidata processing logic
7f84df2
@hellais
Add historical-geoip submodule for testing netinfodb
d50a9d3
@hellais
Add timestamp and observation_id to nettest observations
8cf9d23
@hellais
Update create_tables script
cb704af
@hellais
Make torinfo optional
3c83b6a
@hellais
Add support for passing in a country code
383426d
@hellais
black format
c3617a5
@hellais
Merge branch 'dnscheck' into more-observations
ad115f6 
* dnscheck:
  Improvements based on feedback from @bassosimone
@bassosimone
Merge branch 'main' into more-observations
153fc74
@hellais
Add fast fail argument
050f94b
@hellais
Write fail log to file
d224ddb
@hellais
Merge branch 'more-observations' of github.com:ooni/data into more-ob…
db197a1 
…servations

* 'more-observations' of github.com:ooni/data:
@hellais
Add support for country based verdict generation
454c454
@hellais
Improvements to the dns consistency checks
297c2d6
@hellais
Place bound on consistency ranges
3b44cf3
@hellais
Debugging of verdict generation
9b10a8f
@hellais
Add probe_cc filter for domains in a day
b475c9c
@hellais
Use ASN for consistency check
c89a8f2
@hellais
Add DNS consistency confidence value
7b19a9c
@hellais
Add tls consistency baseline table for situations when we don't have it
f1f98c4 
* Don't lookup the metadata of bogon IPs or non IP DNS answers
@hellais
Provide type hints via strings to avoid circular import
768f0ae
@hellais
ipaddress.ip_address actually raises ValueError
d86f54d
@hellais
Use sets for discaring missing answers
5890d17
@hellais
Add support for parallelism
42e35c5
@hellais
Add support for writing to a runtime log
728dac0
@hellais
Fix return of imap_unordered
74552b2
@hellais @bassosimone
More observations (#3)
015cbc0 
* Fix tests for netinfodb

* Update create tables

* Improvements to the oonidata processing logic

* Add historical-geoip submodule for testing netinfodb

* Add timestamp and observation_id to nettest observations

* Update create_tables script

* Make torinfo optional

* Add support for passing in a country code

* black format

* Add fast fail argument

* Write fail log to file

Co-authored-by: Simone Basso <bassosimone@gmail.com>
@hellais
Improve the spec of dataformats
e15ffcf
@hellais
Improvements to observationts
254d4a8 
* Use report_id instead of session_id
* Add input to base observation
* Extract richer resolver metadata
* Add function for generating observations given a WebConnectivity
measurement
@hellais
Refactor processing code
c1bd90a 
* Improve ergonomics of CLI arguments
* Don't parallise on CSV generation
* Use observation generator for web_connectivity
@hellais
Store report_id and input inside of Verdicts
70057e9
@hellais
Merge remote-tracking branch 'origin/main' into mashumaro
356888e 
* origin/main:
  fix(dataformat): network_events.transaction_id is Optional[int] (#5)
  More observations (#3)
@hellais
Drop bench_loadmsmt
9250533 
This has been moved over here: https://github.com/hellais/oonidata-bench
@hellais
Add support for specifying start_day and end_day as strings
a64fb2b
@hellais
Make it soft fail when removing the probes measurement from the baseline
d17cffa
@hellais
Parallelise the get_file_entries listing
98dda7d
@hellais
Add progress callback on prefix listing
934cac2
@hellais
Add test for progress_callback on listing
3fb8580
@hellais
Fix issue with decompressing old postcans
d91e43e
@hellais
Switch to ThreadPool
e0fc9c8
@hellais
Refactor the progress updates and include the file entry count in the…
7112636 
… listing
@hellais
Set default probe_network_name
e733875
@hellais
Post a progress update at download begin
01cd191
@hellais
Update progress callback
7bcb0fe
@hellais
Make resolver metadata optional
5c9a1e9
@hellais
Properly count the total file size
73627f0
@hellais
Move calculation of total size out of function
1855a95
@hellais
Return the number of bytes read in the current file entry instead of …
48d049f 
…the total
@hellais
Add comment about bad tls.mitm heuristics
01d30c6
@hellais
Add support for generating baselines from control measurements
b53f199
@hellais
Add support for loading whatsapp test
e137dd2
@hellais
Add support for passing file_entries list to allow parallel processing
810167b
@hellais
Cast probe string to int
17f517d
@hellais

hellais commented Oct 18, 2022

Copy link
Copy Markdown
Member Author

I am moving forward with self merging this so I can work on a integrating the new historical-geoip DB
@hellais
hellais merged commit 51c697d into main Oct 18, 2022
@hellais
hellais deleted the mashumaro branch October 18, 2022 14:34
@hellais hellais mentioned this pull request Mar 15, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

funder/isoc2022

2 participants

@hellais @bassosimone