From the course: Kafka Essentials: Quick Start for Building Effective Data Pipelines by Pearson
Join today to access over 25,200 courses taught by industry experts.
Understanding partition allocation and management - Kafka Tutorial
From the course: Kafka Essentials: Quick Start for Building Effective Data Pipelines by Pearson
Understanding partition allocation and management
In previous lessons, we talked about partition allocation and a little bit about management. I want to cover some deeper topics here in this lesson to round out your understanding of Kafka partitions. Let's start with an example. Suppose you have six brokers and create a topic with 10 partitions and a replication That means Kafka has 30 factor of three. 10, since we're using 10 partitions. A few notes. So Kafka clusters should have a minimum of three brokers. You can work as we have done in Analytics Hadoop minimal with just one broker to understand how Kafka works, but in any kind of production situation you should have at least three brokers in the event one broker should fail. You'll have at least two others and a quorum of systems to to run from. A good rule is to set the number of partitions to the number of brokers and the replication factor of three. Now, in this case, because we're demonstrating how Kafka allocates partitions, we're going to use oversubscribe, basically, the…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.