Skip to content

Evaluate different data format for speeding up reprocessing #59

Open

Description

@hellais

Reprocessing is currently not so fast, given that it's feeding from JSONL files. We should consider using something that's more performant like avro, msgpack or similar.

For historical purposes, here are some benchmarks I ran sometime ago while building OONI Data:

Activity

  1. self-assigned this
    on Mar 15, 2024
  2. added
    enhancementNew feature request or improvement to existing functionality
    on Mar 15, 2024
  3. hellais commented on Jan 23, 2025

    @hellais
    MemberAuthor

    As part of this we should also consider the use of seekable Zstandard (see:ooni/backend#782) and some of the considerations part of: ooni/backend#191

  4. moved this to Backlog in Roadmapon Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions