Skip to main content

Velero - Cluster backups and disaster recovery

Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

Velero consists of:

  • A server that runs on your cluster
  • A command-line client that runs locally

Velero CLI Install:

brew install velero

Velero is installed on all clusters as part of the cloud-platform-components install. The module is installed by this section of the code.

Velero is installed in namespace:velero using helm. The chart used can be found here

You can verify the Velero server pod is installed and running using the kubectl get pods command:

kubectl get pods -n velero

All backups are are stored in a S3 bucket for 30 days. To view the backup location run the following command:

velero get backup-locations

The output will show the bucket name and corresponding prefix.

Backups

Why?

There are hundreds of reasons why you need to backup parts and/or your full cluster, a few high level reasons are:

  • The cluster can not be recovered back to to a fully working state after a failed upgrade
  • A user accidentally deletes a resource or namespace
  • Losing or corrupted persistent storage (EBS Volume)

What?

Velero can be used to back up as little as a single resource type to the whole cluster (all namespaces and resources).

StatefulSets including corresponding persistent volumes are backed up together, as unlike stateless applications, it is not possible to easily get an application restored when it has persistent data.

Velero does not backup the Kubernetes state stored in etcd. It is highly recommenced that a specific backup solution is used for etcd and its relevant certificates in addition to Velero.

How?

It is possible to create manual backups as well as schedules ones.

Manual backups

velero backup create my-app-backup01 --include-namespaces home

The above command will create a backup called my-app-backup01 and will backup all resources within the home namespace.

If you accidentally deleted namespace:home, you can recover the namespace and all resources by running the following command:

velero restore create --from-backup my-app-backup01

Scheduled backups

Currently there is one scheduled backup created as part of the default install. This schedule is called velero-allnamespacebackup. This backup includes all namespaces, resources and persistent volumes (EBS) associated with StatefulSets. The Schedule is set to run every 3 hours.

To view all stored backups within the cluster, run the following command:

velero get backups

It is also possible to restore (and exclude) specific namespaces and resources from a velero-allnamespace backup. Below are examples of restoring a specific namespace and resource.

velero restore create --from-backup velero-allnamespacebackup --include-namespaces monitoring
velero restore create --from-backup velero-allnamespacebackup --include-resource prometheusrules

In a disaster recovery scenario, scheduled backups continue to backup, this may create backups with incorrect config. Instead of stopping individual scheduled jobs, it is recommended to change the storage location access mode to ReadOnly. To do this you require the storage location name. the following command will give you the storage location name and current access mode:

velero get backup-locations

Then running the below:

kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
    --namespace velero \
    --type merge \
    --patch '{"spec":{"accessMode":"ReadOnly"}}'

Once the disaster recovery scenario is resolved, you can set storage location mode back to ReadWrite using the following command:

kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
       --namespace velero \
       --type merge \
       --patch '{"spec":{"accessMode":"ReadWrite"}}'

Point Cluster B to to backup folder of Cluster A

It is possible to restore resources from backups from another cluster. You need to add a new Backup Storage Location and Volume Snapshot Location within the primary cluster pointing it to the backup cluster folder you want to restore from.

velero backup-location create <location-name> --provider aws --bucket <bucket-name> --prefix <clusterprefix>  --access-mode=ReadOnly
 velero snapshot-location create <location-name> --provider aws

At this point, when you execute velero backups get you will see 2 different locations under the storage location column.

If you want to create normal backups with 2 different storage locations configured, you either have to set one of the locatons as default. or add --volume-snapshot-locations to your backup command.

velero backup create <backup-name> --volume-snapshot-locations <location-name>

Click here for the official Velero documentation.

This page was last reviewed on 23 June 2021. It needs to be reviewed again on 23 September 2021 by the page owner #cloud-platform .
This page was set to be reviewed before 23 September 2021 by the page owner #cloud-platform. This might mean the content is out of date.