Kubernetes – restore a failed single to multi-master cluster migration

This guide will help you restore your Kubernetes failed migration from single to multi-master.

Using Kubernetes cluster deployed with kops on Amazon EC2.

First of all you need to have in handy your etcd and etcd-events backups.

Example of how to backup:

a – Backup main etcd cluster

$ kubectl --namespace=kube-system get pods | grep etcd
etcd-server-events-ip-172-20-36-161.ec2.internal        1/1       Running   4          2h
etcd-server-ip-172-20-36-161.ec2.internal               1/1       Running   4          2h
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -it -- sh
/ # etcdctl backup --data-dir /var/etcd/data --backup-dir /var/etcd/backup
/ # mv /var/etcd/backup/ /var/etcd/data/
/ # exit
$ kubectl --namespace=kube-system get pod etcd-server-ip-172-20-36-161.ec2.internal -o json | jq '.spec.volumes[] | select(.name | contains("varetcdata")) | .hostPath.path'
"/mnt/master-vol-0ea119c15602cbb57/var/etcd/data"
$ ssh admin@<master-node>
admin@ip-172-20-36-161:~$ sudo -i
root@ip-172-20-36-161:~# mv /mnt/master-vol-0ea119c15602cbb57/var/etcd/data/backup /home/admin/
root@ip-172-20-36-161:~# chown -R admin: /home/admin/backup/
root@ip-172-20-36-161:~# exit
admin@ip-172-20-36-161:~$ exit
$ scp -r admin@<master-node>:backup/ .

b – Backup event etcd cluster

$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -it -- sh
/ # etcdctl backup --data-dir /var/etcd/data-events --backup-dir /var/etcd/backup
/ # mv /var/etcd/backup/ /var/etcd/data-events/
/ # exit
$ kubectl --namespace=kube-system get pod etcd-server-events-ip-172-20-36-161.ec2.internal -o json | jq '.spec.volumes[] | select(.name | contains("varetcdata")) | .hostPath.path'
"/mnt/master-vol-0bb5ad222911c6777/var/etcd/data-events"
$ ssh admin@<master-node>
admin@ip-172-20-36-161:~$ sudo -i
root@ip-172-20-36-161:~# mv /mnt/master-vol-0bb5ad222911c6777/var/etcd/data-events/backup/ /home/admin/backup-events
root@ip-172-20-36-161:~# chown -R admin: /home/admin/backup-events/
root@ip-172-20-36-161:~# exit
admin@ip-172-20-36-161:~$ exit
$ scp -r admin@<master-node>:backup-events/ .

 

Restore (if migration to multi-master failed)

In case you failed to upgrade to multi-master you will need to restore from the backup you have taken previously.

Take extra care becase kops will not start etcd and etcd-events with the same ID on an/or for example but will mix them (ex: etcd-b and etcd-events-c on & etcd-c and etcd-events-b on ); this can be double checked in Route53 where kops will create DNS records for your services.

If your 2nd spinned master failed and cluster becomes inconsistent edit the coresponding kops master instancegroup and switch MinSize and MaxSize to “0” and run an update on your cluster.

Next ssh into your primary master:

systemctl stop kubelet systemctl stop protokube

Reinitialize the etcd instances:

  • In both /etc/kubernetes/manifests/etcd-events.manifest and /etc/kubernetes/manifests/etcd.manifest, add the ETCD_FORCE_NEW_CLUSTER variable with value 1.
  • Delete the containers and the data directories while restoring also from previous backup:
root@ip-172-20-116-230:~# docker stop $(docker ps | grep "gcr.io/google_containers/etcd" | awk '{print $1}')
root@ip-172-20-116-230:~# rm -r /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/member/
root@ip-172-20-116-230:~# rm -r /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/member/
root@ip-172-20-116-230:~# cp -R /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/backup/member  /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/
root@ip-172-20-116-230:~# cp -R /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/backup/member /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/

Now start back the services and watch for the logs:

systemctl start kubelet tail -f /var/log/etcd* # for errors, if no errors encountered re-start also protokubesystemctl start protokube

Test if your master is reboot-proof:

Go to EC2 and Terminate the instance and check if your cluster recovers (needed to discard any manual configurations and check that kops handles everything the right way).

Note! Would recommend also to use Amazon Lambda to take daily Snapshots of all your persistent volume so you can have from what to recover in case of failures.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s