Using RabbitMQ in Microservices

Since Masstransit support was added for RabbitMQ, and the switch from MSMQ happend, I fall in love with RabbitMq, and I was proposing its usage in all my projects. Nowadays, I was challenged with a need of proper setup of RabbitMq in a Microservices context, where the cluster is built around the idea it discovers its nodes using same method I'm using in the service discovery implementation (Consul). finally it becomes official, I can just prepare the docker image of rabbitMQ to enable service discovery, and using ansible along with terraform I can provision the cluster nodes as part of Docker Swarm provisioning pipeline cycle.

Initially, I have a rabbitMQ node on each node, where I utilizes a fact that each container with published port (or using host network) is accessible through Docker bridge IP (172.17.0.1), and on my service container I configure extra hosts record to map that IP to "rabbit" so I can connect to using "rabbit://admin@rabbit".

Good times don't last long!while being in production, we faced issue with nodes are experiencing issues, and disconnect from cluster due to network partitioning, I know that RabbitMQ don't survive among network portioning, and to solve that kind of issues additional steps are needed, but this is different topic for another article. my focus inhere is to ensure that my queues are still accessible and my services don't break.

hhmmm, what can solve the issue??? first thing came to my mind is to have high available policy for the queues along with round-robin setup for DNS among all cluster IP(s), but this doesn't scale well in our datacenter, additionally, it couldn't understand the failures on rabbitMQ so that node is avoided when DNS is resolved.

but....., I found it......, consul is there! I configured the service to lookup rabbit using consul DNS, yes that was a code change, but it solve my issue, it boosts the resiliency and scalability of the whole system.... Win Win (^∇^)