Migrating Kubernetes Clusters with Velero

Velero can be used to migrate and restore clusters. Migrating to a new cluster is slightly more complicated as you have to bootstrap the minimum resources necessary and install velero correctly to perform a restore. Velero has instructions for migration here: https://velero.io/docs/master/migration-case/.

Installing Velero to an existing cluster

First install the velero client with brew:

brew install velero

There are a few options you have for installing velero to an existing cluster. https://velero.io/docs/master/install-overview/. You can also install the velero helm chart. https://github.com/helm/charts/tree/master/stable/velero. You will need to setup an aws bucket and iam role and pass those into velero. You will then need to add a backup schedule for velero which will send backups to the configured bucket. Confirm that you have buckets being created and synchronized to your aws bucket: velero backup get

Bootstrapping a new cluster with kops

First use kops to create a new cluster identical to the cluster you are migrating from: https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_cluster.md

kops create cluster \
--cloud aws \
--image "KOPS_IMAGE" \
--target=terraform \
--kubernetes-version=KUBE_VERSION\
--out=./terraform \
--name=kube-test.domainname

To boot up your cluster apply the changes in the terraform directory.

Once this is done, you will need to have kops export the kubeconfig specs for you to log into the new cluster:

kops export kubecfg kube-test.domainname

You should now have a new cluster you can access by switching your kube context.

Installing Velero to the new cluster

When you setup velero in your original cluster, velero created a secret called cloud-credentials in the velero namespace. You will need to extract the AWS creds to be able to access the velero backup bucket in your new cluster. Set your kubecontext to the cluster you are migrating from and extract this secret.

kubectl get secret cloud-credentials -n velero -o yaml — export > secret.txt

Switch your context back to the the new cluster, and run the following:

velero install \
--secret-file=secret.txt \
--provider aws \
--backup-location-config region=AWS_REGION \
--snapshot-location-config region=AWS_REGION \
--plugins=velero/velero-plugin-for-aws:v1.0.0 \
--bucket VELERO_BUCKET

At the time of this writing, it is required to include a plugin when doing an install. You will need to set the VELERO_BUCKET to the bucket where your backups are being stored to from your original cluster. The snapshot-location-config region is needed to ensure that persistent volumes can be created by velero, else you will get an error where velero restores the persistent volume claims but does not create persistent volumes for them.

Finally remove the secret file rm secret.txt

Check that velero is up and running:

kubectl get pod -n velero

Check that you can see the existing backups for your existing cluster

velero backup get

Restoring the backup:

In your new cluster, pick out the snapshot you need with velero backup get or just pick the latest snapshot and create a restore from it:

velero restore create — from-backup `velero backup get | awk ‘FNR==2 {print $1}’` — exclude-namespaces “external-dns,velero”.

Check that the backup went according to plan by describing the backup. Additionally make sure velero has generated persistent volumes for you when it restored your PVC’s.

Removing the Cluster

To remove your cluster run kops delete cluster CLUSTERNAME — yes. Allow enough time for kops to remove resources.

Notes

In the backup, you exclude the external-dns and velero namespaces. External-dns is a popular kubernetes add-on that will update route53 entries with your ingress routes. Including external-dns will cause all route53 entries for ingresses that external-dns manages to be overwritten to point to the new test cluster instead of your old cluster. Including velero will clash with your existing velero installation.

Your new velero installation uses the same backup location as your old cluster, you may wish to create new settings with velero backup-location create and velero snapshot-location-create and or create schedules for backups with the new cluster. Note if you want to override the default setting you will need to delete the CRD and recreate it: kubectl delete BackupStorageLocation default -n velero and kubectl deleteVolumeSnapshotLocation default -n velero