The age old question from an operations perspective is: ‘Do we have a backup strategy for the company?’  If the answer is yes, then we would also ask ‘does that strategy include accommodations for emerging technologies such as Kubernetes?’ We believe this question comes to bear in a couple of different ways when it pertains to Kubernetes native infrastructure such as Red Hat OpenShift. In today's day and age of cloud computing, the pervasive answer is to recreate. However, every organization does not always have the strategy to recreate the resources fresh and new. We believe there is a mix of “recreate new” versus “backup and restore.” One such option comes to mind, and that is an open source project call velero. Velero is a project started by a company called Heptio, recently purchased by Dell/VMware, that is capable of backing up all the resources of a project as well as backing up the persistent storage used by the applications. For the duration of the article we will focus on the implementation of that solution and what it looks like in a practical way using examples from a real cluster. A word of caution to the reader: Velero has a support matrix that correlates to a minimum version of Kubernetes, which can be found here.

Architecture

Our architecture is a client-server based setup. As we progress through this document we will explore how to setup the environment with all the needed components as well as show you how to create a backup of a project/namespace and restore that very backup to the same cluster, but it could easily target another cluster.

Prerequisites

Now let’s move to the install phases. Velero as a solution has some prerequisites for installation and configuration as you will see listed below. For the purposes of this article we will assume an OpenShift Cluster is used as the primary platform to install and run our Velero and S3 compatible components.

  • Must have an OCP cluster >= 3.7 (Due to the CRD support requirement).
  • Logged in as a cluster-admin to configure all the components.
  • S3-Compatible Object store.
    • In our examples we show you how to install minio, an S3 compatible object store, as well as how to install and use an AWS S3 object store.

 

We will use the velero Github repository to start our install process by first cloning the repo locally and then deploying all the base prerequisites such as CRD’s, a namespace/project, a service account and a cluster role binding using the provided 00-prereqs.yaml file.

$ git clone https://github.com/heptio/velero.git

$ oc create -f velero/examples/common/00-prereqs.yaml

Install Minio (An on-prem s3 solution )

Now that you have the project created and some prerequisites for the project; lets deploy mino, an S3 object store. Our setup will be very basic just to prove out the solution and store our Kubernetes objects.

$ oc apply -f velero/examples/minio/00-minio-deployment.yaml

 

Let’s expose the route so you can login to the web UI using the default user and password in our basic configuration of minio/minio123. This setting and others may be modified in the deployment configuration 00-minio-deployment.yaml.

 

Let’s take advantage of one of the OpenShift features using a route to expose minio to the world.

$ oc expose service/minio -n velero

Adding Persistent Storage via NFS Shares found here:  

** note a bucket of /velero is created with the current deployment, but when we set the storage volume that goes away and should be recreated.

 

There are 2 persistent mounts that need to be patched: As we will see the name and the path match in naming, we created a pvc to match in naming as well. Please ensure we have back end storage already setup, such as NFS, and exported to the cluster. Once that is setup we will need to create a pv (persistent volume) and pvc (persistent volume claim) and ensure they are bound. In the example below, the claim-name(s) reference a pvc name we created in this example. Now let’s set the volumes after our deployment.

$ oc set volume deployment.apps/minio --add --overwrite --name=storage \

       --mount-path=/storage --type=persistentVolumeClaim  \

       --claim-name="minio-pvc-storage"

$ oc set volume deployment.apps/minio --add --overwrite --name=config \

       --mount-path=/config --type=persistentVolumeClaim  \

       --claim-name="minio-pvc-config"

** Note: You can login to the Web UI with the credentials used above and add a Velero bucket.

Install AWS’ s3 Bucket instead of minio

Create the S3 Bucket & IAM User (AWS):

$ aws s3api create-bucket \
   --bucket <YOUR_BUCKET> \
   --region <YOUR_REGION> \
   --create-bucket-configuration LocationConstraint=<YOUR_REGION>

 

$ aws iam create-user --user-name velero

$ BUCKET=<YOUR_BUCKET>
cat > velero-policy.json <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "ec2:DescribeVolumes",
               "ec2:DescribeSnapshots",
               "ec2:CreateTags",
               "ec2:CreateVolume",
               "ec2:CreateSnapshot",
               "ec2:DeleteSnapshot"
           ],
           "Resource": "*"
       },
       {
           "Effect": "Allow",
           "Action": [
               "s3:GetObject",
               "s3:DeleteObject",
               "s3:PutObject",
               "s3:AbortMultipartUpload",
               "s3:ListMultipartUploadParts"
           ],
           "Resource": [
               "arn:aws:s3:::${BUCKET}/*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3:::${BUCKET}"
           ]
       }
   ]
}
EOF

aws iam put-user-policy \
 --user-name velero \
 --policy-name velero \
 --policy-document file://velero-policy.json

 

Create access key

$ aws iam create-access-key --user-name velero

 

The output will contain the SecretAccessKey and AccessKeyId

{
   "AccessKey": {
         "UserName": "velero",
         "Status": "Active",
         "CreateDate": "2019-02-28T13:37:00.000Z",
         "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
         "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
}

 

Make note of the SecretAccessKey and AccessKeyId as you will be unable to retrieve them again. Create the file credentials-velero in your local directory:

[default]

aws_access_key_id=<AWS_ACCESS_KEY_ID>

aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

Installation of Velero

 

We will use the example of the deployment that resides in the examples directory for minio, with no modifications to the yaml, to deploy velero onto our OpenShift cluster.

$ oc create -f velero/examples/minio/20-deployment.yaml

Define backup location

Now that the Velero server is up, it’s time to create a default location for our backup objects. In this case the location described in the yaml file has a name default. You can also create several backup locations to be utilized by creating a copy of this file and modify the metadata section as well as the spec section of the configuration.

These additional backup locations can be other S3 Buckets or even another S3 compatible object storage entirely.

If using aws replace <YOUR_BUCKET> and <YOUR_REGION> in velero/examples/aws/05-backupstoragelocation.yaml

 

Minio AWS
$ oc create -f velero/examples/minio/05-backupstoragelocation.yaml $ oc create -f velero/examples/aws/05-backupstoragelocation.yaml

$ oc create -f - << EOF
---

apiVersion: velero.io/v1

kind: VolumeSnapshotLocation

metadata:

 name: aws-default

 namespace: velero

spec:

 provider: aws

 config:

region: <YOUR_REGION>

...
EOF

# Next we Import the aws credentials as secret into the OpenShift velero namespace.

$ oc create secret generic cloud-credentials \

--namespace velero \

--from-file cloud=credentials-velero

 

Backup

Using the Velero cli

As we walk through the examples together, please know that OpenShift is an enterprise option for Kubernetes and has components that don’t exist in the open source project such as s2i,  build configs, deployment configs, and image streams; however, those objects are also backed up using Velero. Due to the nature of s2i builds in this example, we are excluding the builds, replication controller and pods objects.

$ velero backup create test-project-s2i --include-namespaces test --exclude-resources build.build.openshift.io,replicationcontroller,pods

 

Without the Velero cli

We want to show how to do it without the Velero CLI. In an enterprise environment, not everyone can download binaries from the internet and put them on their machines. However, if you already have OpenShift running you can just create kubernetes native objects. See the example below

$ oc create -f - << EOF
---apiVersion: velero.io/v1

kind: Backup

metadata:

 name: test-velero

 namespace: velero

 labels:

   velero.io/storage-location: default

spec:

 includedNamespaces:

 - test

 excludedResources:

 - build.build.openshift.io

 - replicationcontroller

 - pods

 storageLocation: default

 ttl: 720h0m0s

...
EOF

 

As you can see both objects were created using each method.

$ oc get backups

NAME                   AGE

test-project-s2i       2m

test-velero            56s

Testing time!

Now that we have a backup of our project. In this case a PHP front end using a MySQL database:

$ oc get pods -n test

NAME                            READY STATUS RESTARTS AGE

cakephp-mysql-example-2-8wpx9   1/1 Running 1 18d

mysql-1-4ml6m                   1/1 Running 2 18d

 

It’s time to remove the project:

$ oc delete project test

project.project.openshift.io "test" deleted

 

Prove it’s gone:

$ oc get pods -n test

No resources found.

$ oc get project test

No resources found.

Error from server (NotFound): namespaces "test" not found

 

Recovery using the Velero CLI

Time to create it from the backup:

$ velero restore create --from-backup test-project-s2i

Restore request "test-project-s2i-20190225183058" submitted successfully.

Run `velero restore describe test-project-s2i-20190225183058` or `velero restore logs test-project-s2i-20190225183058` for more details.

 

Recovery without the Velero CLI

We learned how to create a backup without the CLI and using Kubernetes native tooling.

The same can be done when scheduling recovery of an application.

Using this method grants you more granular control over the recovery process.

Following an example:

$ oc create -f - << EOF
---apiVersion: velero.io/v1

kind: Restore

metadata:

 name: test-project-s2i-20190225183058

 namespace: velero

spec:

 backupName: test-project-s2i

 excludedResources:

 - nodes

 - events

 - events.events.k8s.io

 - backups.ark.heptio.com

 - backups.velero.io

 - restores.ark.heptio.com

 - restores.velero.io

 includedNamespaces:

 - test

...

EOF

 

Cleanup

There is additional cleanup required with service accounts and builds. Don’t worry, this is not dangerous: OpenShift will recreate these for you due to the way s2i works. Once your resources are created you will be able to clean up the service accounts and builds. You will also be able to start a new build using the buildconfig name.

$ oc delete sa builder deployer default

$ oc delete build --all

$ oc start-build cakephp-mysql-example

Conclusion

We originally posed the question: “Do we have a backup strategy for the company pertaining to emerging technologies such as Kubernetes?”  

 

The answer is a yes and as the project matures through wider adoption and progresses into deployment options like the operator framework it will ensure the install and configuration are seamless.

 

As with any open source project, velero is evolving. As of the writing of this article we overcame a name/project change from ark to velero. We also encountered how to deal with objects that are specific to OpenShift. One feature we did not discuss is the backup of the clusters persistence data in the light of a Kubernetes persistent volume using an additional open source project called restic and velero’s native support for it. This will be covered in Part 2 of this article…..


About the author

Red Hatter since 2018, tech historian, founder of themade.org, serial non-profiteer.

Read full bio