feat(kafka-deduplicator): integrate BatchConsumer into processing pipeline #40760
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eli-r-ph
commented
Oct 31, 2025
| branches: | ||
| - 'master' | ||
| - 'eli.r/kdedup-batch-consumer-inst-1' | ||
|
|
There was a problem hiding this comment.
☝️ will remove this after smoke testing
eli-r-ph
commented
Oct 31, 2025
| pub original_headers: Option<OwnedHeaders>, | ||
|
|
||
| pub original_payload: Option<Vec<u8>>, | ||
| } |
There was a problem hiding this comment.
☝️ makes these easier to pass along - just grabbing the stuff we need after message hydration instead of a full msg detach() 👍
![greptile-apps[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
eli-r-ph
force-pushed
the
eli.r/kdedup-batch-consumer-inst-1
branch
from
9ac0101 to
c9ef324
Compare
eli-r-ph
force-pushed
the
eli.r/kdedup-batch-consumer-inst-1
branch
from
2307e0e to
c9f4637
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Before fully replacing the
StatefulConsumerwe want to smoke test theBatchConsumeron live event stream.Changes
Just enough refactor to get the
BatchConsumerrunning in production. Due toMessageProcessortrait and more elaborate test plumbing required by stateful consumer to integrate with offset tracker, this isn't a complete/clean cutover yet.The goal here is to deploy the change and surface any unhandled
KafkaErrorsor other informational events that might be causing the batch consumer to shut down for no reason as was observed in an earlier smoke test.Once that is sorted out, we can integrate smarter batch decomposition and routing to processing pool, and a proper outgoing health check, and ship the new consumer 🚀
How did you test this code?
Locally, CI, and (soon) smoke test on live event stream
👉 Stay up-to-date with PostHog coding conventions for a smoother review.
Changelog: (features only) Is this feature complete?
No update required ✅