Blog

Using Ceph as storage for Kubernetes clusters

Wouldn’t be a nice feature to use Rados Block Devices for Kubernetes persistent storage?
Most of the steps used to setup a basic Kubernetes cluster are presented here and will detail only the Ceph integration part.

Presuming that Kubernetes cluster is up and running jumping to the next steps:

Creating the rbd volume

On cephadmin node:

rbd create kube_ceph -s 10240 --image-feature layering

rbd map kube_ceph

mkfs.xfs /dev/rbd0

rbd unmap kube_ceph

 

Preparing Kubernetes to embrace Ceph

 

On any ceph node:

ceph auth get-key client.admin

echo "AQDT5ppYCuM3ChAAXOkw8h4ii7qsnybdXV9D1g==" | base64 > secret.txt

 

On Kubernetes master node:

vi ceph-secret.yml

apiVersion: v1
kind: Secret
metadata:
   name: ceph-secret
data:
   key: < cat secret.txt here >

 

vi ceph-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata: 
   name: ceph-pv
spec:
   capacity:
      storage: 10Gi
   accessModes:
      – ReadWriteMany
   rbd: 
      monitors: 
         – knode01:6789
         – knode02:6789
         – knode03:6789
     pool: rbd
        image: kube_ceph
        user: admin
        secretRef: 
           name: ceph-secret
        fsType: xfs
        readOnly: false

 

vi ceph-pv-claim.yml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
   name: ceph-claim
spec:
   accessModes:
      – ReadWriteMany
   resources:
      requests:
      storage: 10Gi

 

kubectl create -f ceph-secret.yml

kubectl get secret

You should receive a similar output:

[root@master ~]# kubectl get secret
NAME TYPE DATA AGE
ceph-secret Opaque 1 1d
default-token-cgstk kubernetes.io/service-account-token 3 2d

 

kubectl create -f ceph-pv.yml

kubectl create -f ceph-pv-claim.yml

 

Now let’s create a MySQL pod with Ceph Persistent Storage

vi myqsl-pod.yml

apiVersion: v1
kind: Pod
metadata:
   name: ceph-mysql
spec:
   containers:
      – name: ceph-mysql
         image: tutum/mysql
      ports:
         – name: mysql-db
            containerPort: 3306
      volumeMounts:
         – name: mysql-pv
           mountPath: /var/lib/mysql
   volumes:
      – name: mysql-pv
         persistentVolumeClaim:
            claimName: ceph-claim

 

kubectl create -f mysql-pod.yml

 

And now you can enjoy your new MySQL container that has Ceph backed storage.

Zabbix HA Monitoring with Galera and HAProxy

Zabbix is another enterprise grade open source monitoring tool that has the advantage to be MySQL/PostgreSQL/SQLite/Oracle backed for storing
all the configurations and collected data.

As an alternative to installing agent on hosts it includes support for SNMP, TCP and ICMP checks, also JMX, SSH, Telnet, IPMI.

List of interesting features:

  • High performance, high capacity
  • Auto-discovery support
  • Distributed monitoring
  • Web based interface
  • Agent and Agent-less support

Setup prerequisites:

  • 2 LXC containers for Zabbix instances
  • Pre-installed and working Galera cluster
  • Pre-installed and working HAProxy cluster

 

Steps to create Zabbix HA:

 

Install Zabbix on both containers

for host in {zabbix01,zabbix02}; do ssh root@$host 'rpm -ivh http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm'; done

for host in {zabbix01,zabbix02}; do ssh root@$host 'yum -y install zabbix-server-mysql zabbix-web-mysql'; done

 

Create database user on the Galera cluster

create database zabbix;
grant all privileges on zabbix.* to zabbix@'192.168.%' identified by 'Password123';

 

Populate the database with initial configuration

zcat /usr/share/doc/zabbix-server-mysql-3.0.0/create.sql.gz | mysql -u root -p zabbix -h galera.domain.tld

 

Configure Zabbix

vi /etc/zabbix/zabbix_server.conf

and change the lines below to march your Galera informations

DBHost=galera.domain.tld
DBName=zabbix
DBUser=zabbix
DBPassword=Password123

 

Enable and start Zabbix

for host in {zabbix01,zabbix02}; do ssh root@$host 'systemctl enable zabbix && systemctl start zabbix'; done

for host in {zabbix01,zabbix02}; do ssh root@$host 'systemctl enable httpd && systemctl start httpd'; done

 

Avoid deadlocks because of HAProxy and Zabbix internals

Login to your HAProxy cluster nodes and make sure you are not using round robin load balancing but active-backup for Galera nodes

vi /etc/haproxy/haproxy.cfg

And change it to be similar to the setup below:

# Zabbix web frontend
listen http *:80
         option httplog
         option httpclose
         option forwardfor
         reqadd X-Forwarded-Proto:\ http

# define hosts
acl zabbix hdr(host) -i zabbix.domain.tld

# figure out what backends to use based on request
use_backend zabbix_cluster if zabbix

backend zabbix_cluster
          balance roundrobin
          server zabbix01 192.168.100.30:80 check
          server zabbix02 192.168.100.31:80 check

# Galera backend
listen galera *:3306
         mode tcp
         option httpchk
         balance leastconn
         server galera01 192.168.100.101:3306 check port 9200
         server galera02 192.168.100.102:3306 check port 9200 backup
         server galera03 192.168.100.103:3306 check port 9200 backup

 

For HAProxy health check on Galera nodes go http://www.zeding.ro/galera-cluster-for-mysql-true-multi-master-on-centosrhel-7-x/ at paragraph HAProxy monitoring for Galera.

 

Restart HAProxy

systecmtl restart haproxy

 

Now access you Zabbix setup

Point your browser to http://zabbix.domain.tld

Default credentials are: Admin / zabbix

Now you can start customizing and using it.

LXC systemd-journal process eats CPU on CentOS/RHEL 7.x

Sometimes systemd-journal process eats CPU and locks system load to 2.00 which is abnormal operation and in order to fix this there is a trick

Set “lxc.kmsg = 0” in the container config
vi /var/lib/lxc/dns01/config

and add

lxc.kmsg = 0

Start container if not already started and from within run:
rm -f /dev/kmsg

Restart lxc service & container
systemctl restart lxc.service
lxc-stop –name=dns01 && lxc-start –name=dns01

LXC attach extra block device to container on CentOS/RHEL 7.x

You might ask why do I need an extra block device?
Most of the time you do not need it but special cases might occur and I’ll give an example: we want to test GlusterFS with distributed/replicated bricks using plain block devices inside containers.

1st make sure you have installed lxc
yum install lxc lxc-templates

Setup the container
lxc-create --template=centos --name=gluster01 --bdev=lvm --vgname=cached_raid1 --fssize=4G

Create 2 new extra block devices
lvcreate -V 10G --thin -n gluster01_brick1 raid_10/thin_pool_secondary
lvcreate -V 10G --thin -n gluster01_brick2 raid_10/thin_pool_secondary

Grab minor and major of the new block device
ls -al /dev/mapper/raid_10-gluster01_brick1
ls -al /dev/mapper/raid_10-gluster01_brick2

Configure the container with the new block device
vi /var/lib/lxc/gluster01/config

add

# Mounting extra disk
lxc.aa_profile = lxc-container-default-with-mounting
lxc.cgroup.devices.allow = b 253:37 rwm # 1st brick
lxc.cgroup.devices.allow = b 253:38 rwm # 2nd brick
lxc.autodev = 1
lxc.hook.autodev = /var/lib/lxc/gluster01/autodev_block.sh

Create /var/lib/lxc/gluster01/autodev_block.sh
with the following content

#!/bin/bash
cd ${LXC_ROOTFS_MOUNT}/dev
mknod -m 666 sdb b 253 37
mknod -m 666 sdc b 253 38

then make it executable

chmod +x /var/lib/lxc/gluster01/autodev_block.sh

Start the container and check everything
lxc-start --name=gluster01

and run inside the container:

fdisk -l

You should see 2 disks of 10GB.

libvirt usefull commands – Part 1

In each part will present 5 usefull libvirt commands that are not very popular.

  • Increase memory
  • Attach an extra disk
  • Extra network interface
  • Take screenshot
  • Change vCPU number

 

Increase memory

virsh setmaxmem <vm_name> 1024M --config && virsh setmem <vm_name> 1024M --config

 

Attach an extra disk

virsh attach-disk gluster01 /dev/raid_0/gluster01_brick1 vdb --cache none --config 

 

Extra network interface

virsh attach-interface --domain <vm_name> --type network --source <network> --model virtio --mac 12:23:34:45:56:67 --config

 

Take screenshot

virsh screenshot <vm_name>

 

Change vCPU number

virsh setvcpus <vm_name> 4 --config

Highly Available RabbitMQ messaging cluster

What is RabbitMQ? Robust messaging for applications; it is easy to use, runs on all major OSes, open source and commercially supported.

Why HA? For redundancy, in case one fails would like to preserve the queues.

This time I will take another approach and deploy, configure with Ansible.

On your ansible machine make sure you have the following 2 files and 2 LXC containers already installed (DNS records included).

[root@ansible01 ansible]# ls
files hosts tasks

[root@ansible01 ansible]# less hosts

[rabbitmqs]
rabbitmq01.example.com
rabbitmq02.example.com

[root@ansible01 tasks]# ls
install-rabbitmq-cluster.yml

[root@ansible01 tasks]# less install-rabbitmq-cluster.yml

---
#
# Config RabbitMQ Cluster
#
- hosts: rabbitmqs

sudo: no

tasks:

- name: Config rabbitmq01 basic openstack packages
yum: name=centos-release-openstack-mitaka state=latest

- name: Force a yum upgrade
shell: yum -y upgrade

- name: Install python-openstackclient
yum: name=python-openstackclient state=latest

- name: Install openstack-selinux
yum: name=openstack-selinux state=latest

- name: Install mariadb
yum: name=mariadb state=latest

- name: Install python2-PyMySQL
yum: name=python2-PyMySQL state=latest

- name: Install EPEL repo
yum: name=epel-release state=latest

- name: Install net-tools - reprequisites for RabbitMQ
yum: name=net-tools state=latest

- name: Install Erlang - prerequisite for RabbitMQ
yum: name=python2-PyMySQL state=latest

- name: Install RabbitMQ
yum: name=rabbitmq-server state=latest

- name: Start RabbitMQ
shell: systemctl enable rabbitmq-server

- hosts: rabbitmq01.example.com

sudo: no

tasks:

- name: Copy erlang cookie to secondary node
shell: scp /var/lib/rabbitmq/.erlang.cookie root@rabbitmq02.example.com:/var/lib/rabbitmq/.erlang.cookie
- hosts: rabbitmqs

sudo: no

tasks:

- name: Start RabbitMQ
shell: systemctl start rabbitmq-server
- hosts: rabbitmq02.example.com

sudo: no

tasks:

- name: Join secondary node to cluster - stage 1
shell: rabbitmqctl stop_app

- name: Join secondary node to cluster - stage 2
shell: rabbitmqctl join_cluster --ram rabbit@rabbitmq01

- name: Join secondary node to cluster - stage 3
shell: rabbitmqctl start_app

- hosts: rabbitmq01.example.com

sudo: no

tasks:

- name: Ensure that all queues are mirrored across all running nodes
shell: rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'

 

Then you execute the playbook like this:

ansible-playbook ansible/tasks/install-rabbitmq-cluster.yml -i /root/ansible/hosts --ask-pass

 

And Voila! your RabbitMQ cluster is UP.

HAProxy active/backup load balancer setup on CentOS/RHEL 7.x

In these days everyone wants to use High Availability and this is a good thing, so no more single points of failure.
Will demonstrate how this can be achieved using the following:
– lxc containers
– haproxy
– pacemaker
– corosync
– CentOS 7.x

Steps to follow:

If you are using lxc containers, “which” is not installed by default and it is needed
yum -y install which

Install pacemaker and pcs
yum -y install pacemaker pcs

Enable and restart the services
systemctl enable pcsd.service && systemctl start pcsd.service
systemctl enable pacemaker.service && systemctl start pacemaker.service
systemctl enable corosync.service && systemctl start corosync.service

Make sure you remove fqdn record from line containing "127.0.0.1" in /etc/hosts

Add node01 and node02 to /etc/hosts or as DNS records
vi /etc/hosts

And add:

192.168.100.2 node01
192.168.100.3 node02

Setup a password for hacluster user
passwd hacluster

Very important: permit traffic in firewalld
firewall-cmd --add-service=high-availability && \
   firewall-cmd --permanent --add-service=high-availability

Authentication between cluster members and cluster creation
pcs cluster auth node01 node02
pcs cluster setup --name haproxy_cluster node01 node02
pcs cluster start --all

Nodes must be restarted before moving on!

Check everything is right
pcs status
pcs status corosync
pcs status nodes

Disable stonith or your resources will not start
pcs property set stonith-enabled=false

Create the VIP resource for HAProxy
pcs resource create virtual_ip_haproxy ocf:heartbeat:IPaddr2 ip=192.168.100.100 cidr_netmask=24 op monitor interval=10s

Check again if the resource has started
pcs status resources

And:

ip a sh

Setup bash alias to copy haproxy.cfg to secondary node before restarting the service on primary
vi ~/.bashrc

And add:

alias haproxy_restart="haproxy -f /etc/haproxy/haproxy.cfg -c && \
    scp /etc/haproxy/haproxy.cfg root@node02:/etc/haproxy/haproxy.cfg && sleep 5 && systemctl restart haproxy"

Get haproxy ocf resource from web
cd /usr/lib/ocf/resource.d/heartbeat && \
    curl -O https://raw.githubusercontent.com/thisismitch/cluster-agents/master/haproxy && \
    chmod +x haproxy

Add resource for haproxy in cluster
pcs resource create haproxy ocf:heartbeat:haproxy op monitor interval=10s

Create a clone of haproxy resource that must run on both nodes
pcs resource clone haproxy

Now everything should be right, the only thing that remains to configure is /etc/haproxy.cfg and it is not the scope of this post.