From the course: Apache Kafka Essential Training: Building Scalable Applications

Unlock this course with a free trial

Join today to access over 25,300 courses taught by industry experts.

Multi-threaded consumers

Multi-threaded consumers - Kafka Tutorial

From the course: Apache Kafka Essential Training: Building Scalable Applications

Start my 1-month free trial Buy for my team

Multi-threaded consumers

“

- In the previous video, we have the consumer running in a single thread and processing messages one by one. Processing a single message can take significant time depending upon the use case. On the other hand messages can be produced at much higher speeds and the single thread may not catch up. We can start multiple consumers in a consumer group, but even there, a single consumer needs to at least handle a single partition which may be quite large. To optimize resource utilization, consumers can dequeue messages in one thread and then use multiple threads to process them in parallel. The code here is a simple example on how this can be achieved. This is not the most optimal nor production grade multi-threaded code. It is just a sample to demonstrate one scalability option. It is highly recommended to focus on advanced Java capabilities for optimal multi-threading. The sample code is available in the…

Contents