Blog

Dynamic service discovery on top of Kubernetes – part 2 (self-registering etcd GO service)

This is part 2 of the article; an example of self-registering etcd service with key:value expiration if no backed renewal triggered.

You need to have the etcd independent deployment from previous article (part 1) running, find it https://zeding.ro/2018/09/08/dynamic-service-discovery-on-top-of-kubernetes-part-1-etcd-cluster/ or https://www.linkedin.com/pulse/dynamic-service-discovery-top-kubernetes-part-1-etcd-victor-laza/

Written in GO, the service is a simple pong JSON

#server.go
package main
import (
  "net/http"
  "net"
  "github.com/gin-gonic/gin"
  "os"
  "strings"
)

func main() {

	r := gin.Default()

	name, _ := os.Hostname()
	addresses, _ := net.LookupHost(name)

	msg := []string{"pong from", addresses[0]}
	r.GET("/ping", func(c *gin.Context) {
		c.JSON(http.StatusOK, gin.H{"message": strings.Join(msg, " ")})
	})

	// Listen and serve on 0.0.0.0:80
	r.Run(":80")
}


#register.go
package main
import (
	"strings"
	"net/http"
	"time"
)

func main() {
        // register
        body := strings.NewReader(`value=${MY_POD_IP}:80&ttl=5`)
        req, err := http.NewRequest("PUT", "http://etcd-client-service:2379/v2/keys/pong-service/backends/${HOSTNAME}", body)
        if err != nil {
        // handle err
        }
        req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
        // handle err
        }
        defer resp.Body.Close()

	// refresh loop
	for {
	// refresh ttl
        body := strings.NewReader(`ttl=5&refresh=true&prevExist=true`)
        req, err := http.NewRequest("PUT", "http://etcd-client-service:2379/v2/keys/pong-service/backends/${HOSTNAME}", body)
        if err != nil {
        // handle err
        }
        req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
        // handle err
        }
        defer resp.Body.Close()
        time.Sleep(time.Second * 4)
	}
}


The 2 services have to be integrated in a Docker image and ran in parallel (supervisord would be an option).

Already created image can be found here: https://hub.docker.com/r/zeding/alpine-pong/tags/

If you want to run it on Kubernetes also bellow is the YAML template.

#pong.yaml
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: pong
spec:
  replicas: 2
  revisionHistoryLimit: 15
  template:
    metadata:
      labels:
        app: pong
    spec:
      containers:
        - name: pong
          - env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
                  image: zeding/alpine-pong:1.0.0
          ports:
            - containerPort: 80

Create the deployment:

kubectl -n <your-namespace> create -f pong.yaml

And now you have a functional etcd self-registering GO service.

Here is what you should have in your namespace:

  kubectl -n <your-namespace> get pods -o wide

  NAME                    READY     STATUS              RESTARTS   AGE       IP              NODE
  infra-etcd-cluster-0    1/1       Running             0          1d        100.96.13.85    ip-172-20-42-48.eu-central-1.compute.internal
  infra-etcd-cluster-1    1/1       Running             0          1d        100.96.15.148   ip-172-20-44-47.eu-central-1.compute.internal
  infra-etcd-cluster-2    1/1       Running             0          1d        100.96.10.144   ip-172-20-43-86.eu-central-1.compute.internal
  pong-5fd5578995-6f9bk   1/1       Running             0          24s       100.96.7.220    ip-172-20-44-102.eu-central-1.compute.internal
  pong-5fd5578995-rz7h8   0/1       ContainerCreating   0          24s       <none>          ip-172-20-44-212.eu-central-1.compute.internal

Live monitoring your backends from etcd:

watch "etcdctl ls --recursive -p | grep -v '/$' | xargs -n 1 -I% sh -c 'echo -n %:; etcdctl get %;' | cut -d ':' -f2"

example output ** multiple endpoints ** IPs only:

100.96.7.220
100.96.7.218

Now you can create a script to fetch those from etcd and use for dynamic containers communications.

Dynamic service discovery on top of Kubernetes – part 1 (etcd cluster)

This article will show you how to provision a custom etcd cluster for service discovery on top of Kubernetes; it is unrelated with the default etcd that drives Kubernetes.

Can be used to store key/value pair and for dynamic service that will register to it, live based on TTL settings.

Article presents “part 1”, the creation of the isolated etcd cluster.

Prerequisites – environment as tested

Kubernetes: 1.8.15

Cloud provider: Amazon

Docker containers: Alpine Linux 3.8

Golang: 1.11

KubeDNS: pre-configured

Public personal repository for Docker images: zeding/*

Create all yaml templates

You will end up with 3 files in a folder called test-etcd

# service.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: infra-etcd-cluster
    app: etcd
  name: infra-etcd-cluster
spec:
  clusterIP: None
  ports:
  - name: infra-etcd-cluster-2379
    port: 2379
    protocol: TCP
    targetPort: 2379
  - name: infra-etcd-cluster-2380
    port: 2380
    protocol: TCP
    targetPort: 2380
  selector:
    k8s-app: infra-etcd-cluster
    app: etcd
  type: ClusterIP

# service-client.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: etcd-client-service
    app: etcd
  name: etcd-client-service
spec:
  ports:
  - name: infra-etcd-cluster-2379
    port: 2379
    protocol: TCP
    targetPort: 2379
  selector:
    k8s-app: infra-etcd-cluster
    app: etcd
  sessionAffinity: None
  type: ClusterIP

# etcd-cluster.yaml 
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  labels:
    k8s-app: infra-etcd-cluster
    app: etcd
  name: infra-etcd-cluster
spec:
  replicas: 3
  selector:
    matchLabels:
      k8s-app: infra-etcd-cluster
      app: etcd
  serviceName: infra-etcd-cluster
  template:
    metadata:
      labels:
        k8s-app: infra-etcd-cluster
        app: etcd
      name: infra-etcd-cluster
    spec:
      containers:
      - command:
        - /bin/sh
        - -ec
        - |
          HOSTNAME=$(hostname)
          echo "etcd api version is ${ETCDAPI_VERSION}"

          eps() {
              EPS=""
              for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
                  EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379"
              done
              echo ${EPS}
          }

          member_hash() {
              etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | cut -d':' -f1 | cut -d'[' -f1
          }

          initial_peers() {
                PEERS=""
                for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
                PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380"
                done
                echo ${PEERS}
          }

          # etcd-SET_ID
          SET_ID=${HOSTNAME##*-}
          # adding a new member to existing cluster (assuming all initial pods are available)
          if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
              export ETCDCTL_ENDPOINTS=$(eps)

              # member already added?
              MEMBER_HASH=$(member_hash)
              if [ -n "${MEMBER_HASH}" ]; then
                  # the member hash exists but for some reason etcd failed
                  # as the datadir has not be created, we can remove the member
                  # and retrieve new hash
                  if [ "${ETCDAPI_VERSION}" -eq 3 ]; then
                      ETCDCTL_API=3 etcdctl --user=root:${ROOT_PASSWORD} member remove ${MEMBER_HASH}
                  else
                      etcdctl --username=root:${ROOT_PASSWORD} member remove ${MEMBER_HASH}
                  fi
              fi
              echo "Adding new member"
              rm -rf /var/run/etcd/*
              # ensure etcd dir exist
              mkdir -p /var/run/etcd/
              # sleep 60s wait endpoint become ready
              echo "sleep 60s wait endpoint become ready,sleeping..."
              sleep 60

              if [ "${ETCDAPI_VERSION}" -eq 3 ]; then
                  ETCDCTL_API=3 etcdctl --user=root:${ROOT_PASSWORD} member add ${HOSTNAME} --peer-urls=http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | grep "^ETCD_" > /var/run/etcd/new_member_envs
              else
                  etcdctl --username=root:${ROOT_PASSWORD} member add ${HOSTNAME} http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | grep "^ETCD_" > /var/run/etcd/new_member_envs
              fi



              if [ $? -ne 0 ]; then
                  echo "member add ${HOSTNAME} error."
                  rm -f /var/run/etcd/new_member_envs
                  exit 1
              fi

              cat /var/run/etcd/new_member_envs
              source /var/run/etcd/new_member_envs

              exec etcd --name ${HOSTNAME} \
                  --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 \
                  --listen-peer-urls http://0.0.0.0:2380 \
                  --listen-client-urls http://0.0.0.0:2379 \
                  --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379 \
                  --data-dir /var/run/etcd/default.etcd \
                  --initial-cluster ${ETCD_INITIAL_CLUSTER} \
                  --initial-cluster-state ${ETCD_INITIAL_CLUSTER_STATE}
          fi

          for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
              while true; do
                  echo "Waiting for ${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE} to come up"
                  ping -W 1 -c 1 ${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE} > /dev/null && break
                  sleep 1s
              done
          done

          echo "join member ${HOSTNAME}"
          # join member
          exec etcd --name ${HOSTNAME} \
              --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 \
              --listen-peer-urls http://0.0.0.0:2380 \
              --listen-client-urls http://0.0.0.0:2379 \
              --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379 \
              --initial-cluster-token etcd-cluster-1 \
              --data-dir /var/run/etcd/default.etcd \
              --initial-cluster $(initial_peers) \
              --initial-cluster-state new

        env:
        - name: INITIAL_CLUSTER_SIZE
          value: "3"
        - name: CLUSTER_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: ETCDAPI_VERSION
          value: "3"
        - name: ROOT_PASSWORD
          value: "Password123"
        - name: SET_NAME
          value: "infra-etcd-cluster"
        - name: GOMAXPROCS
          value: "4"
        image: zeding/alpine-etcd:3.3.9
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -ec
              - |
                HOSTNAME=$(hostname)

                member_hash() {
                    etcdctl member list | grep http://${HOSTNAME}.${SET_NAME}.${CLUSTER_NAMESPACE}:2380 | cut -d':' -f1 | cut -d'[' -f1
                }

                eps() {
                    EPS=""
                    for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
                        EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${CLUSTER_NAMESPACE}:2379"
                    done
                    echo ${EPS}
                }

                export ETCDCTL_ENDPOINTS=$(eps)

                SET_ID=${HOSTNAME##*-}
                # Removing member from cluster
                if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
                    echo "Removing ${HOSTNAME} from etcd cluster"
                    if [ "${ETCDAPI_VERSION}" -eq 3 ]; then
                        ETCDCTL_API=3 etcdctl --user=root:${ROOT_PASSWORD} member remove $(member_hash)
                    else
                        etcdctl --username=root:${ROOT_PASSWORD} member remove $(member_hash)
                    fi
                    if [ $? -eq 0 ]; then
                        # Remove everything otherwise the cluster will no longer scale-up
                        rm -rf /var/run/etcd/*
                    fi
                fi
        name: infra-etcd-cluster
        ports:
        - containerPort: 2380
          name: peer
          protocol: TCP
        - containerPort: 2379
          name: client
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
  updateStrategy:
    type: OnDelete


Provisioning the new etcd-cluster

cd test-etcd

kubectl -n test-namespace create -f .

 

Test you new cluster

kubectl -n test-namespace get pods -o wide

        NAME                    READY     STATUS              RESTARTS   AGE       IP              NODE
        infra-etcd-cluster-0    1/1       Running             0          1d        100.96.13.85    ip-172-20-42-48.eu-central-1.compute.internal
        infra-etcd-cluster-1    1/1       Running             0          1d        100.96.15.148   ip-172-20-44-47.eu-central-1.compute.internal
        infra-etcd-cluster-2    1/1       Running             0          1d        100.96.10.144   ip-172-20-43-86.eu-central-1.compute.internal


In the following “part 2” will present a custom pong service written in Go that will self-register to etcd and update available/healthy backends.

Kubernetes – restore a failed single to multi-master cluster migration

This guide will help you restore your Kubernetes failed migration from single to multi-master.

Using Kubernetes cluster deployed with kops on Amazon EC2.

First of all you need to have in handy your etcd and etcd-events backups.

Example of how to backup:

a – Backup main etcd cluster

$ kubectl --namespace=kube-system get pods | grep etcd
etcd-server-events-ip-172-20-36-161.ec2.internal        1/1       Running   4          2h
etcd-server-ip-172-20-36-161.ec2.internal               1/1       Running   4          2h
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -it -- sh
/ # etcdctl backup --data-dir /var/etcd/data --backup-dir /var/etcd/backup
/ # mv /var/etcd/backup/ /var/etcd/data/
/ # exit
$ kubectl --namespace=kube-system get pod etcd-server-ip-172-20-36-161.ec2.internal -o json | jq '.spec.volumes[] | select(.name | contains("varetcdata")) | .hostPath.path'
"/mnt/master-vol-0ea119c15602cbb57/var/etcd/data"
$ ssh admin@<master-node>
admin@ip-172-20-36-161:~$ sudo -i
root@ip-172-20-36-161:~# mv /mnt/master-vol-0ea119c15602cbb57/var/etcd/data/backup /home/admin/
root@ip-172-20-36-161:~# chown -R admin: /home/admin/backup/
root@ip-172-20-36-161:~# exit
admin@ip-172-20-36-161:~$ exit
$ scp -r admin@<master-node>:backup/ .

b – Backup event etcd cluster

$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -it -- sh
/ # etcdctl backup --data-dir /var/etcd/data-events --backup-dir /var/etcd/backup
/ # mv /var/etcd/backup/ /var/etcd/data-events/
/ # exit
$ kubectl --namespace=kube-system get pod etcd-server-events-ip-172-20-36-161.ec2.internal -o json | jq '.spec.volumes[] | select(.name | contains("varetcdata")) | .hostPath.path'
"/mnt/master-vol-0bb5ad222911c6777/var/etcd/data-events"
$ ssh admin@<master-node>
admin@ip-172-20-36-161:~$ sudo -i
root@ip-172-20-36-161:~# mv /mnt/master-vol-0bb5ad222911c6777/var/etcd/data-events/backup/ /home/admin/backup-events
root@ip-172-20-36-161:~# chown -R admin: /home/admin/backup-events/
root@ip-172-20-36-161:~# exit
admin@ip-172-20-36-161:~$ exit
$ scp -r admin@<master-node>:backup-events/ .

 

Restore (if migration to multi-master failed)

In case you failed to upgrade to multi-master you will need to restore from the backup you have taken previously.

Take extra care becase kops will not start etcd and etcd-events with the same ID on an/or for example but will mix them (ex: etcd-b and etcd-events-c on & etcd-c and etcd-events-b on ); this can be double checked in Route53 where kops will create DNS records for your services.

If your 2nd spinned master failed and cluster becomes inconsistent edit the coresponding kops master instancegroup and switch MinSize and MaxSize to “0” and run an update on your cluster.

Next ssh into your primary master:

systemctl stop kubelet systemctl stop protokube

Reinitialize the etcd instances:

  • In both /etc/kubernetes/manifests/etcd-events.manifest and /etc/kubernetes/manifests/etcd.manifest, add the ETCD_FORCE_NEW_CLUSTER variable with value 1.
  • Delete the containers and the data directories while restoring also from previous backup:
root@ip-172-20-116-230:~# docker stop $(docker ps | grep "gcr.io/google_containers/etcd" | awk '{print $1}')
root@ip-172-20-116-230:~# rm -r /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/member/
root@ip-172-20-116-230:~# rm -r /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/member/
root@ip-172-20-116-230:~# cp -R /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/backup/member  /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/
root@ip-172-20-116-230:~# cp -R /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/backup/member /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/

Now start back the services and watch for the logs:

systemctl start kubelet tail -f /var/log/etcd* # for errors, if no errors encountered re-start also protokubesystemctl start protokube

Test if your master is reboot-proof:

Go to EC2 and Terminate the instance and check if your cluster recovers (needed to discard any manual configurations and check that kops handles everything the right way).

Note! Would recommend also to use Amazon Lambda to take daily Snapshots of all your persistent volume so you can have from what to recover in case of failures.

Kubernetes – RBAC limited multi-namespace administrator

This page will guide you into creating namespace administrator on Kubernetes with RBAC enabled; you need to be a platform administrator and have access to api.

The example below is for an administrator with multi-namespace access.

Create the certificates

openssl genrsa -out multi-namespace-admin.key 2048
openssl req -new -key multi-namespace-admin.key \
-out multi-namespace-admin.csr \
-subj "/CN=multi-namespace-admin/O=namespace-admins"
openssl x509 -req -in multi-namespace-admin.csr \
-CA ca.crt \
-CAkey ca.key -CAcreateserial \
-out multi-namespace-admin.crt -days 365

 

Set credentials and contexts

kubectl config set-credentials multi-namespace-admin \
--client-certificate=multi-namespace-admin.crt \
--client-key=multi-namespace-admin.key
kubectl config set-context multi-admin-demo-context \
--cluster=api.cluster.com \
--namespace=demo \
--user=multi-namespace-admin
kubectl config set-context multi-admin-sandbox-context \
--cluster=api.cluster.com \
--namespace=sandbox \
--user=multi-namespace-admin

 

Create roles and rolebindings

You need to create one such pair of files for each namespace.

# role-multi-namespace-manager.yml 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
   namespace: demo
   name: multi-namespace-manager
rules:
   - apiGroups: ["", "extensions", "apps"]
     resources: ["pods"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
kubectl create -f role-multi-namespace-manager.yml
# rolebinding-multi-namespace-manager.yml 
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
   name: multi-namespace-manager-binding
   namespace: demo
subjects:
 - kind: User
   name: multi-namespace-admin
   apiGroup: ""
roleRef:
   kind: Role
   name: multi-namespace-manager
   apiGroup: ""
kubectl create -f rolebinding-multi-namespace-manager.yml 

Create the config file for kubectl

# .kube/config
apiVersion: v1
clusters:
- cluster:
   certificate-authority-data: XXXXX
   server: https://api.cluster.com
 name: api.cluster.com
contexts:
- context:
   cluster: api.cluster.com
   namespace: demo
   user: multi-namespace-admin
 name: multi-admin-demo-context
- context:
   cluster: api.cluster.com
   namespace: sandbox
   user: multi-namespace-admin
 name: multi-admin-sandbox-context
current-context:
kind: Config
preferences: {}
users:
- name: multi-namespace-admin
 user:
   client-certificate: multi-admin.crt
   client-key: multi-admin.key

Usage

In order to access a namespace where you have permissions change the context

 

kubectl --context=multi-admin-demo-context get pods --namespace=demo
NAME                                       READY    STATUS   RESTARTS  AGE
default-http-backend-2198840601-pmc9k      1/1      Running  1         89d
nginx-ingress-controller-4182399137-mb5l6  1/1      Running  0         3d
nginx-ingress-controller-4182399137-xb8bk  1/1      Running  0         3d
pgsql                                      1/1      Running  0         69d
tomcat-597660569-8bhqr                     1/1      Running  1         27d

 

 

kubectl --context=multi-admin-sandbox-context get pods --namespace=sandbox
NAME                                       READY    STATUS   RESTARTS  AGE
default-http-backend-2198840601-z61bq      1/1      Running  0         4d
jenkins                                    1/1      Running  0         2d
nginx-ingress-controller-3712311050-1n2z9  1/1      Running  0         4d
nginx-ingress-controller-3712311050-dvpvm  1/1      Running  0         4d
repo                                       1/1      Running  0         5d

 

Below are some examples where you do not have permissions.

 

kubectl --context=multi-admin-demo-context get pods --namespace=kube-system
Error from server (Forbidden): the server does not allow access to the requested resource (get pods)

kubectl --context=multi-admin-test-context get pods --namespace=test
error: context "multi-admin-test-context" does not exist

Enjoy your new limited multi-namespace administrator account.

Kubernetes – RabbitMQ HA cluster

This is a short how-to that describes the way to setup an RabbitMQ cluster with auto-healing inside specified Kubernetes namespace.

For the sake of simplicity I will create only one yaml that includes both services and statefulset definitions.

Preparations

But first things first, we need to create an Erlang cookie and save it as Kubernetes secret.

cat /dev/urandom | tr -dc ‘a-zA-Z0-9’ | fold -w 64 | head -n 1

kubectl –context=cluster-name -n sandbox create secret generic rabbitmq-config –from-literal=erlang-cookie=<the 64 string>

Now create kubernetes-rabbitmq-cluster.yaml

---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-management
labels:
app: rabbitmq
spec:
ports:
- port: 15672
name: http
selector:
app: rabbitmq
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
labels:
app: rabbitmq
spec:
ports:
- port: 5672
name: amqp
- port: 4369
name: epmd
- port: 25672
name: rabbitmq-dist
clusterIP: None
selector:
app: rabbitmq
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: rabbitmq
spec:
serviceName: "rabbitmq"
replicas: 3
template:
metadata:
labels:
app: rabbitmq
spec:
containers:
- name: rabbitmq
image: rabbitmq:3.6.6-management-alpine
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- >
if [ -z "$(grep rabbitmq /etc/resolv.conf)" ]; then
sed "s/^search \([^ ]\+\)/search rabbitmq.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
cat /etc/resolv.conf.new > /etc/resolv.conf;
rm /etc/resolv.conf.new;
fi;
until rabbitmqctl node_health_check; do sleep 1; done;
if [[ "$HOSTNAME" != "rabbitmq-0" && -z "$(rabbitmqctl cluster_status | grep rabbitmq-0)" ]]; then
rabbitmqctl stop_app;
rabbitmqctl join_cluster rabbit@rabbitmq-0;
rabbitmqctl start_app;
fi;
rabbitmqctl set_policy ha-all "." '{"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic"}'
env:
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: rabbitmq-config
key: erlang-cookie
ports:
- containerPort: 5672
name: amqp

kubectl –context=cluster-name -n sandbox create -f kubernetes-rabbitmq-cluster.yaml

Now check your new cluster

kubectl –context=cluster-name -n sandbox exec rabbitmq-0 rabbitmqctl cluster_status

Cluster status of node 'rabbit@rabbitmq-0' ...
[{nodes,[{disc,['rabbit@rabbitmq-0','rabbit@rabbitmq-1',
                'rabbit@rabbitmq-2']}]},
 {running_nodes,['rabbit@rabbitmq-2','rabbit@rabbitmq-1','rabbit@rabbitmq-0']},
 {cluster_name,<<"rabbit@rabbitmq-0.rabbitmq.sandbox.svc.cluster.local">>},
 {partitions,[]},
 {alarms,[{'rabbit@rabbitmq-2',[]},
          {'rabbit@rabbitmq-1',[]},
          {'rabbit@rabbitmq-0',[]}]}]

Kubernetes & Jenkins – on-demand Selenium Grid

This is a fast-forward guide for testers using Kubernetes, Jenkins & Selenium Grid

Prerequisites:

  • A running Kubernetes cluster with internal DNS working
  • A running Jenkins container
  • A proper defined “selenium-hub:4444(TCP)” service inside the namespace

Define a variable “SELENIUM” as Boolean Parameter; default value is up to you.

 

Add the following to your build stage in Jenkins and make sure you are using parametrised builds.

#!/bin/bash

if [[ '$SELENIUM'='ENABLED' ]]
then
echo "Spinning Selenium Grid.."
SELENIUM_YAML=$(mktemp)
cat <<EOF >> $SELENIUM_YAML
---
apiVersion: v1
kind: Pod
metadata:
name: selenium-hub
labels:
app: selenium-hub
spec:
containers:
- name: selenium-hub
image: selenium/hub
ports:
- containerPort: 4444 
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /grid/console
port: 4444
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /grid/console
port: 4444
initialDelaySeconds: 30
timeoutSeconds: 5
---
apiVersion: v1
kind: Pod
metadata:
name: selenium-node-chrome
labels:
app: selenium-node-chrome
spec:
containers:
- name: selenium-node-chrome
image: selenium/node-chrome
env:
- name: HUB_PORT_4444_TCP_ADDR
value: "selenium-hub"
- name: HUB_PORT_4444_TCP_PORT
value: "4444"
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "250m"
---
apiVersion: v1
kind: Pod
metadata:
name: selenium-node-firefox
labels:
app: selenium-node-firefox
spec:
containers:
- name: selenium-node-firefox
image: selenium/node-firefox
env:
- name: HUB_PORT_4444_TCP_ADDR
value: "selenium-hub"
- name: HUB_PORT_4444_TCP_PORT
value: "4444"
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "250m"
EOF
kubectl create -f $SELENIUM_YAML
rm -f $SELENIUM_YAML  
else
echo "Selenium Grid not enabled!"
fi

 

As Post-build step in Jenkins job add the following; it can be added also if you are using cascading jobs, sky’s the limit.

#!/bin/bash
if [[ '$SELENIUM'='ENABLED' ]]
then
echo "Spinning down Selenium Grid.. Testing is done"
kubectl delete pod selenium-hub selenium-node-chrome selenium-node-firefox
else
echo "Selenium Grid was not enabled - Nothing to be done.."
fi

Kubernetes – Deploy Nginx Ingress Controller as layer between your applications and external Load Balancers

Everyone knows what Kubernetes is, most of us know how to expose a service in NodePort, ClusterIP or LoadBalancer mode but not everyone knows how to use Ingress.

What is an Ingress Controller? In short it is a Load Balancer.

Why do we need it? For more control of you exposed services and not last, cost related; I’ll give an example here: If you are using 100 services exposed via ELBs on Amazon it will cost you around 2,000$/month but if you use one replicated Ingress Controller and expose the 100 services via it it will cost you only 20$/month.

Bellow the order to deploy an Ingress Controller:

Note! This has been tested on Kubernetes 1.6.4

# default-backend.yaml
# kubectl create -f default-backend.yaml
#
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: K8S_NAMESPACE
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
# default-backend-service.yaml
# kubectl create -f default-backend-service.yaml
#
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: K8S_NAMESPACE
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend

# nginx-ingress-controller-conf.yaml
# kubectl create -f nginx-ingress-controller-conf.yaml
#
apiVersion: v1
data:
  enable-sticky-sessions: "true"
  body-size: "5m"
kind: ConfigMap
metadata:
  name: nginx-ingress-controller-conf

# nginx-ingress-controller.yaml
# kubectl create -f nginx-ingress-controller.yaml
#
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-controller
  namespace: K8S_NAMESPACE
spec:
  replicas: 2
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --nginx-configmap=default/nginx-ingress-controller-conf

 

Make sure you have the below DNS records pointing as ALIAS to your ELB on Amazon.


# ingress.yaml
# kubectl create -f ingress.yaml
#
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: customer-ingress
  namespace: K8S_NAMESPACE
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: "default.test-cloud.eu"
    http:
      paths:
      - path: /
        backend:
          serviceName: default-http-backend
          servicePort: 80
  - host: "tomcat.test-cloud.eu"
    http:
      paths:
      - path: /
        backend:
          serviceName: tomcat-service
          servicePort: 8080
# ingress-service.yaml
# kubectl create -f ingress-service.yaml
#
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-service
  namespace: K8S_NAMESPACE
spec:
  type: LoadBalancer
  ports:
    - port: 80
      name: http
    - port: 443
      name: https
  selector:
    k8s-app: nginx-ingress-controller

Give it a try! Enhance your services and reduce your cost overhead.