Kubernetes – RabbitMQ HA cluster

This is a short how-to that describes the way to setup an RabbitMQ cluster with auto-healing inside specified Kubernetes namespace.

For the sake of simplicity I will create only one yaml that includes both services and statefulset definitions.

Preparations

But first things first, we need to create an Erlang cookie and save it as Kubernetes secret.

cat /dev/urandom | tr -dc ‘a-zA-Z0-9’ | fold -w 64 | head -n 1

kubectl –context=cluster-name -n sandbox create secret generic rabbitmq-config –from-literal=erlang-cookie=<the 64 string>

Now create kubernetes-rabbitmq-cluster.yaml

---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-management
labels:
app: rabbitmq
spec:
ports:
- port: 15672
name: http
selector:
app: rabbitmq
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
labels:
app: rabbitmq
spec:
ports:
- port: 5672
name: amqp
- port: 4369
name: epmd
- port: 25672
name: rabbitmq-dist
clusterIP: None
selector:
app: rabbitmq
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: rabbitmq
spec:
serviceName: "rabbitmq"
replicas: 3
template:
metadata:
labels:
app: rabbitmq
spec:
containers:
- name: rabbitmq
image: rabbitmq:3.6.6-management-alpine
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- >
if [ -z "$(grep rabbitmq /etc/resolv.conf)" ]; then
sed "s/^search \([^ ]\+\)/search rabbitmq.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
cat /etc/resolv.conf.new > /etc/resolv.conf;
rm /etc/resolv.conf.new;
fi;
until rabbitmqctl node_health_check; do sleep 1; done;
if [[ "$HOSTNAME" != "rabbitmq-0" && -z "$(rabbitmqctl cluster_status | grep rabbitmq-0)" ]]; then
rabbitmqctl stop_app;
rabbitmqctl join_cluster rabbit@rabbitmq-0;
rabbitmqctl start_app;
fi;
rabbitmqctl set_policy ha-all "." '{"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic"}'
env:
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: rabbitmq-config
key: erlang-cookie
ports:
- containerPort: 5672
name: amqp

kubectl –context=cluster-name -n sandbox create -f kubernetes-rabbitmq-cluster.yaml

Now check your new cluster

kubectl –context=cluster-name -n sandbox exec rabbitmq-0 rabbitmqctl cluster_status

Cluster status of node 'rabbit@rabbitmq-0' ...
[{nodes,[{disc,['rabbit@rabbitmq-0','rabbit@rabbitmq-1',
                'rabbit@rabbitmq-2']}]},
 {running_nodes,['rabbit@rabbitmq-2','rabbit@rabbitmq-1','rabbit@rabbitmq-0']},
 {cluster_name,<<"rabbit@rabbitmq-0.rabbitmq.sandbox.svc.cluster.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@rabbitmq-2',[]},
          {'rabbit@rabbitmq-1',[]},
          {'rabbit@rabbitmq-0',[]}]}]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s