Helm is a popular package manager for Kubernetes applications which helps packaging all resources an application needs as a Helm Chart which can then be shared and installed on Kubernetes clusters. Helm Charts are useful for addressing the complexities of installation and simple upgrades of particularly stateless applications like web apps. However when it comes to stateful applications there is more to the upgrade process than upgrading the application itself. If you have ever tried to upgrade a PostgreSQL database before, you know that upgrading PostgreSQL itself is the least of the problems. Dealing with the schema changes between database versions is where the main complexity lies, which often requires dumping the data and re-importing it back to the new version of the database in a controlled manner so that client applications aren’t affected. Therefore, Helm Charts often are insufficient for upgrading stateful applications and services (e.g. PostgreSQL or Elasticsearch) which require a complex and controlled upgrade processes beyond upgrading the application version.
What is Helm Operator?
Operators are a method of packaging and deploying Kubernetes applications that take human operational knowledge of managing the application and encode it into the package itself, which can then be shared with users.
Helm Operator enables pairing a Helm Chart with the operational knowledge of installing, upgrading and managing the application on Kubernetes clusters. The Operator SDK, which is the tool for building operators, can create an operator based on a Helm Chart and essentially allow enriching the Helm Chart capabilities by delivering the expertise of managing and running the application together with the application.
Pairing your existing Helm charts with an Operator is a great way to plug your apps into your Kubernetes cluster and gain Day 2 operations capabilities. This is an update to our previous post about building Helm Operators in 15 minutes. We’ve made it easier, so you can do it in less time and with less work. Stay tuned to the very end to learn how to automate this.
What’s new in the Operator SDK for Helm? An improvement in UX because the Operator SDK now fetches your charts automatically. Below we’re pulling down the stable/mariadb chart:
$ operator-sdk new mariadb-operator \ --type=helm \ --helm-chart stable/mariadb
You can fetch charts from:
- your local disk (great for CI/CD!)
- the Helm “stable” collection of charts
- your custom repo of charts
Once you have a chart for referencing, all you need to do is build the Operator container, and store it in a registry.
Before we begin, make sure you have a working Go environment and have the latest version of the Helm Operator SDK (currently v0.8.1).
Build an Operator from existing chart
The SDK command line tool will scaffold everything required for your Operator, without requiring any code to be written.
$ operator-sdk new mariadb-operator \ --type=helm \ --helm-chart stable/mariadb INFO Creating new Helm operator 'mariadb-operator'. INFO Created helm-charts/mariadb INFO Generating RBAC rules INFO Created build/Dockerfile INFO Created watches.yaml INFO Created deploy/service_account.yaml INFO Created deploy/role.yaml INFO Created deploy/role_binding.yaml INFO Created deploy/operator.yaml INFO Created deploy/crds/charts_v1alpha1_mariadb_crd.yaml INFO Created deploy/crds/charts_v1alpha1_mariadb_cr.yaml INFO Run git init ... INFO Run git init done INFO Project creation complete.
The SDK has scanned our chart, found the minimal set of permissions needed, and generated everything required to install instances of our MariaDB chart.
Next, we need to build and store our Operator container:
$ cd nginx-operator $ operator-sdk build quay.io/robszumski/mariadb-operator:v0.0.1 $ docker push quay.io/robszumski/mariadb-operator:v0.0.1
Later on we will simulate an upgrade to the Operator. Tag the same image as 0.0.2 so we can see that in action.
$ operator-sdk build quay.io/robszumski/mariadb-operator:v0.0.2 $ docker push quay.io/robszumski/mariadb-operator:v0.0.2
That’s it! We have an Operator container ready for installation on our cluster. This Operator represents an immutable artifact of v0.0.1 of our app, which can be consumed by several teams or within CI systems in order to bring up a new deployment of our app in a consistent and repeatable way.
Install the Operator
If you’re using Red Hat OpenShift 4, you can use the pre-installed Operator Lifecycle Manager (OLM) to manage and update Operators on your cluster. The Lifecycle Manager can be installed with one command on any Kubernetes cluster. For non-OLM methods, read below for other ways to deploy your Helm Operator.
First, generate the ClusterServiceVersion that represents the CRDs your Operator uses, the permissions it requires to function and other installation information. You’ll only have to do this once, then carry these changes forward for successive releases of the Operator.
$ operator-sdk olm-catalog gen-csv \ --csv-version 0.0.1 INFO Generating CSV manifest version 0.0.1 INFO Fill in the following required fields in file deploy/olm-catalog/mariadb-operator/0.0.1/mariadb-operator.v0.0.1.clusterserviceversion.yaml: spec.keywords spec.maintainers spec.provider INFO Created deploy/olm-catalog/mariadb-operator/0.0.1/mariadb-operator.v0.0.1.clusterserviceversion.yaml
There are two fields that you need to provide info for: the displayName and the description of your CRD. Fill them in with this command or your customized values:
$ sed -i 's#mariadbs.charts.helm.k8s.io#mariadbs.charts.helm.k8s.io\n displayName: MariaDB\n description: MariaDB config values#g' deploy/olm-catalog/mariadb-operator/0.0.1/mariadb-operator.v0.0.1.clusterserviceversion.yaml
And then replace the image reference with the Quay.io image you just built:
$ sed -i 's#REPLACE_IMAGE#quay.io/robszumski/mariadb-operator:v0.0.1#' deploy/olm-catalog/mariadb-operator/0.0.1/mariadb-operator.v0.0.1.clusterserviceversion.yaml
Next, prepare the namespace we are going to use. Using the openshift-operators namespace is easiest because it is already set up for us to grant access to this Operator for all users of the cluster. Replace the placeholder value in the CSV with our desired namespace:
$ sed -i 's#namespace: placeholder#namespace: openshift-operators#' deploy/olm-catalog/mariadb-operator/0.0.1/mariadb-operator.v0.0.1.clusterserviceversion.yaml $ oc project openshift-operators
Using project “openshift-operators”.
Now complete the installation by submitting the file for the Lifecycle Manager to install and manage:
$ oc create -f deploy/crds/charts_v1alpha1_mariadb_crd.yaml customresourcedefinition.apiextensions.k8s.io/mariadbs.charts.helm.k8s.io created $ oc create -f deploy/service_account.yaml $ oc create -f deploy/role_binding.yaml $ oc create -f deploy/role.yaml $ oc create -f deploy/olm-catalog/mariadb-operator/0.0.1/mariadb-operator.v0.0.1.clusterserviceversion.yaml
You should see a new entry show up in your cluster Console:
If you aren’t on OpenShift 4, or can’t install the Lifecycle Manager on your cluster, you can run the deployment and other files directly, although you won’t benefit from the management capabilities. Install everything in the deploy directory:
$ oc create -f deploy/
Now our Operator is ready to be used by everyone on the cluster.
Deploy an instance of the chart
Here’s the magic of the Operator: we can create many instances of our nginx app by creating new objects, without any special CLIs, in a completely Kubernetes-native way. Create the default object that the SDK generated:
$ oc project default $ oc create -f deploy/crds/charts_v1alpha1_mariadb_cr.yaml
You can also do this in the Console via the Developer Catalog:
Then list all instances running on the cluster:
$ oc get mariadbs NAME AGE example-mariadb 56s
You can also see them in the UI and the resources created by the chart:
Updating to a new version
When a new version of your app is ready, build a new version of the Operator at the same time. This maintains the immutable artifact that can be deployed across your different dev/test/prod environments.
If you used the Operator Lifecycle Manager, you can have the system do a smooth rolling update of your Operator by using the –from-version flag when you generate your CSV. This will update RBAC roles, bindings and service accounts as needed, in addition to using a new version of your chart.
$ operator-sdk olm-catalog gen-csv \ --csv-version 0.0.2 \ --from-version 0.0.1 INFO Generating CSV manifest version 0.0.2 WARN Required csv fields not filled in file deploy/olm-catalog/mariadb-operator/0.0.2/mariadb-operator.v0.0.2.clusterserviceversion.yaml: spec.keywords spec.maintainers spec.provider INFO Created deploy/olm-catalog/mariadb-operator/0.0.2/mariadb-operator.v0.0.2.clusterserviceversion.yaml
You will need to add a reference to the new container image in the CSV:
$ sed -i 's#REPLACE_IMAGE#quay.io/robszumski/mariadb-operator:v0.0.2#' deploy/olm-catalog/mariadb-operator/0.0.2/mariadb-operator.v0.0.2.clusterserviceversion.yaml
Then you are ready to update all of the Operators on the cluster:
$ oc create -f deploy/olm-catalog/mariadb-operator/0.0.2/mariadb-operator.v0.0.2.clusterserviceversion.yaml
All of the existing instances of your app (all of the v1alpha1 MariaDB objects) that exist on the cluster will receive updates based on your chart updates. For example, If you introduced a new component that is defaulted “on,” it would be deployed from the chart. If you changed any existing behavior to make it work better, that change would also take affect across all of the deployed instances. This process can also help when you update your CRD definitions.
After the Operator has updated, you should see that the new version now owns all of the previously deployed MariaDB instances. This is a powerful for making forwards and backwards compatible changes. If other teams have a dependency on your component, they don’t need to be an expert in each version, and can depend on the Operator to handle this correctly.
Operators written in Go and Ansible can handle a sophisticated upgrade, failover and data rebalancing scenarios. Refer to OpenShift documentation in order to read more about creating Operators using Go and Ansible.
Building a delivery pipeline with CI/CD
Plugging this into a testing and CI/CD pipeline is possible because you can reference local charts from disk. If your chart is located within the same repo as your code, you can use it to test both installs and upgrades via the Operator by running the SDK commands we have walked through above.
The next in our blog series about Helm Operators will cover this in more detail.
A note on Helm 2 vs Helm 3
This post covers the Operator SDK v0.8.1, which relies on Helm 2 code in order to process your chart into an Operator. Future versions of the SDK should also be compatible with Helm 3 when it becomes stable. Your usage of the SDK should not have to change and these modifications should be under-the-hood. Stay tuned for more details about this as the Helm 3 effort matures.