From the course: Kafka Essentials: Quick Start for Building Effective Data Pipelines by Pearson
Join today to access over 25,200 courses taught by industry experts.
Understanding hardware choices - Kafka Tutorial
From the course: Kafka Essentials: Quick Start for Building Effective Data Pipelines by Pearson
Understanding hardware choices
In this lesson, I'd like to talk a little bit about some of the choices needed for setting up a Kafka cluster. One of the first questions, though, that gets asked about Kafka is where does Kafka actually store its data? And all Kafka data, Kafka log files, the data that we record with Kafka, are stored locally on the local machine. Kafka data are not stored in Hadoop HDFS or any other distributed file system because that's not quite fast enough for what Kafka wants to do. For each server, the storage location is specified in the local server.properties file in slash etc Kafka, somewhere in that tree. You'll find the server.properties file. And in that file will be a line that says log.dirs. And that will be a list of the locations on the local machine where the data are to be stored. You can use just one location. That's fine. Or you can use multiple. And in the example here, we have slash var slash data one slash Kafka slash Kafka logs. And then we have avar data2 slash Kafka slash…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.