Skip to content

chore(data-import): fix json serialization #40758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore(data-import): fix json serialization #40758

wants to merge 1 commit into from
+66 −1

Conversation

@EDsCODE
Copy link
Collaborator

@EDsCODE EDsCODE commented Oct 31, 2025

Problem

  • the bigquery source directly returns pyarrow tables which haven't serialized all the jsons. This will cause evolve step to fail because evolve uses regular json.dumps

Changes

  • convert to util that uses orjson

How did you test this code?

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Changelog: (features only) Is this feature complete?
@EDsCODE EDsCODE requested a review from a team October 31, 2025 20:17
greptile-apps[bot]
greptile-apps bot reviewed Oct 31, 2025
View reviewed changes
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Comments (1)

  1. posthog/temporal/data_imports/pipelines/pipeline/utils.py, line 482 (link)

    style: bare except catches all exceptions including KeyboardInterrupt and SystemExit

2 files reviewed, 1 comment

Edit Code Review Agent Settings | Greptile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@EDsCODE