From the course: Complete Guide to Apache Kafka for Beginners

Unlock the full course today

Join today to access over 25,200 courses taught by industry experts.

Idempotent producer

Idempotent producer - Kafka Tutorial

From the course: Complete Guide to Apache Kafka for Beginners

Start my 1-month free trial Buy for my team

Idempotent producer

“

So let's review what is an idempotent producer. So when a producer is sending data into Apache Kafka, there can be duplicate messages due to network errors. So let's have an example. Here is what a good request looks like. We produced data into Apache Kafka. Apache Kafka commits the data into the log and Apache Kafka sends back an acknowledgment to our producer. From that point onwards, all good, right? But what if we have a bad request or duplicate request. What happens? We produce data to Kafka. Kafka commits the messages on the log and sends back an ack. But this ack never reaches our producer maybe because of a network error. Therefore, the producer never receives an ack and say this is weird. I'm going to retry my produce because we have a retry setting, right? So the produce is retried. Kafka sees it as a new request, so it will commit a duplicate message and then sends back the ack. So from a producer…

Contents