108 questions
2
votes
1
answer
1k
views
Sarama Kafka ConsumerGroup implementation
I have a decent experience working with kafka and python using confluent kafka library. Recently I had to shift towards golang and use sarama library with it.
And I can not understand how ...
- 101
1
vote
0
answers
105
views
How to prevent Kafka consumers from being prematurely released from a consumer group?
I'm using Apache Kafka(aws msk) with the IBM Sarama Go library to manage a Kafka consumer group. I've encountered an issue where my consumers stop consuming messages under certain conditions, even ...
1
vote
0
answers
139
views
Grant permission to cluster resource kafka use sarama library
I have problem granting permissions to kafka confluent resource cluster using sarama. My code is below, when I run it it gives me an error:
error: kafka server: failed to create one or more ACL rules: ...
1
vote
0
answers
136
views
Kafka version sarama 2.6.0 smooth running on local but in staging error
This is my code I'm using ibm/sarama
version, err := sarama.ParseKafkaVersion(version)
if err != nil {
boot.Logger.L.Critical(err.Error(), err)
log.Panicf("Error parsing ...
- 11
0
votes
0
answers
134
views
cannot create kafka producer with go
I'm trying to create a kafka producer locally. I'm spinning up kafka through docker-compose up like this:
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181&...
- 73
-2
votes
1
answer
238
views
Detecting new partitions in a kafka topic
I am new to Kafka and have a use case where I will be reading data from a kafka topic.
While reading it is possible that new partitions are added. Is there a way with golang Sarama client to detect if ...
- 399
1
vote
1
answer
307
views
ECS Service Can't Connect to MSK Serverless
I have a consumer application written in GoLang that is running as a service in ECS Fargate, which I'm trying to use to consume from an MSK Serverless Cluster. The ECS service is on the same VPC as ...
- 55
2
votes
0
answers
310
views
Commit the offset by manual using sarama library becomes very slow
When I use kafka-producer-perf-test.sh to simulate the producer sending 10,000 messages to a topic 't' with 2 partitions, and use the consumer code mentioned below with dfv1.CommitN set to 3, I ...
- 21
3
votes
1
answer
2k
views
Will sarama client trigger rebalance if this client does not fetch messages for a long time?
As far as I know, there exist two configurations related to the heartbeat timeout and max poll interval in Java kafka consumer, which are:
session.timeout.ms
max.poll.interval.ms
If the interval ...
- 43
0
votes
0
answers
377
views
client/metadata got error from broker -1 while fetching metadata: dial tcp: address tcp/9092": unknown port
there is a problem when I connect to kafka via docker.
this is my docker-compose.yaml
version: '3'
networks:
fixtures_test:
external: true
services:
zookeeper:
image: confluentinc/cp-...
- 1
1
vote
0
answers
476
views
Kafka Go/Sarama and schema registry registration issues with protobuf
I have a compiled proto from another repo that I would like to have registered for an integration test.
I'm basically doing this:
payload := map[string]interface{}{
"schemaType": ...
- 499
1
vote
0
answers
766
views
How to unit test sarama kafka in Go?
I'm trying to unit test kafka client in my library messaging project but got timeout error.
I've copy everything from the sarama library but still got the same timeout error.
I don't know why... can ...
1
vote
0
answers
47
views
Kafka producer "Hello world" in Go [duplicate]
I have 2 docker containers running:
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b92bf8cba740 ...
- 479
1
vote
1
answer
461
views
How can I stop Dapr Kafka Component from connecting to closed consumer groups?
I have the following component in k8s:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub
namespace: default
spec:
type: pubsub.kafka
version: v1
metadata:
# ...
- 127
2
votes
1
answer
667
views
Kafka consumer lag issue when consuming two topics with UTC
I am encountering an issue with my Kafka consumer while consuming two topics (Topic A and Topic B) within a single consumer group. Both topics contain UTC timestamps in their data and are published ...
