Writing a Custom Controller in Python

Have you ever wondered how controllers work in Kubernetes and wished you could make your own?
In this post, I will show you how to use the Python client-kubernetes library to interact with Kubernetes and build your own custom controller interacting with Custom Resource Definitions (CRD).

Requisites

• A running Kubernetes/OpenShift cluster (oc cluster up is fine for that matter)
• Clone my samplecontroller repo if you want to follow the code review

Context

Our custom controller will be watching for Guitar Objects. Upon creation, it will check their brand and put a corresponding comment. Note that the algorithm I use for reviewing guitars is fairly simple, based on my own tastes (and some guitars I own).

Custom Resource Definitions

Those Guitar objects are defined as Custom Resource Definitions, which are a way to provide an abstraction layer on top of etcd and to easily define new objects on top of Kubernetes. They are the successor to ThirdPartyResources (TPRs), though they conceptually provide similar benefits.

This is what a Custom Resource Definition looks like:

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: guitars.kool.karmalabs.local

spec:

  group: kool.karmalabs.local

  version: v1

  scope: Namespaced

  names:

    plural: guitars

    singular: guitar

    kind: Guitar

    shortNames:

     - guit

You can inject this definition with oc create -f and you will then be able to create your own guitar "instances". For instance, to define a stratocaster, you can use the following yaml:

apiVersion: "kool.karmalabs.local/v1"

kind: Guitar

metadata:

  name: stratocaster

spec:

  brand: fender

  review: false

We can populate whatever fields we want in the spec part of the definition, although the custom controller part is needed to make any use of it

Running the Controller

The following commands will:

• Create a dedicated project to hold the controller pod.
• Give enough privileges to this pod (this is needed only because I am checking guitars cluster wide).
• Deploy the custom controller (using the image I created for that matter).

oc new-project guitarcenter

oc adm policy add-cluster-role-to-user cluster-admin -z default -n guitarcenter

oc new-app karmab/samplecontroller

About the Code

Let's review relevant parts of the code:

• By checking whether KUBERNETES_PORT environment variables are defined or not, we detect whether code is being run within a pod or outside. This allows us to use the same code to run manually (which is quite handy during development). At this step, we are authenticated and can talk to the Kubernetes API.

if 'KUBERNETES_PORT' in os.environ:

    config.load_incluster_config()

else:

    config.load_kube_config()

• We check whether the Custom Resource Definition associated to the Guitar kind exists, or otherwise create it using a yaml file containing the same exact content indicated above.

definition = 'guitar.yml'

v1 = client.ApiextensionsV1beta1Api()

current_crds = [x['spec']['names']['kind'].lower() for x in v1.list_custom_resource_definition().to_dict()['items']]

if 'guitar' not in current_crds:

    print("Creating guitar definition")

    with open(definition) as data:

        body = yaml.load(data)

    v1.create_custom_resource_definition(body)

• We enter a loop using a watch to monitor guitar objects. We also wrap the watch with an additional while True to prevent unwanted timeouts.
  I'm using crds.list_cluster_custom_object, instead of methods limited to the current namespace, which is why my controller needs extra permissions.
  Once we get a new object, we check its specs to see if it's been reviewed and if not, and launch an auxiliar method to actually process it.

crds = client.CustomObjectsApi()

DOMAIN = "kool.karmalabs.local"

resource_version = ''

while True:

    stream = watch.Watch().stream(crds.list_cluster_custom_object, DOMAIN, "v1", "guitars", resource_version=resource_version)

    for event in stream:

        obj = event["object"]

        operation = event['type']

        spec = obj.get("spec")

        if not spec:

            continue

        metadata = obj.get("metadata")

        resource_version = metadata['resourceVersion']

        name = metadata['name']

        print("Handling %s on %s" % (operation, name))

        done = spec.get("review", False)

        if done:

            continue

        review_guitar(crds, obj)

How to Use

Create some guitars using the provided yaml files and see the review made for you (notice how the comment field of the spec gets updated).

oc create -f crd/stratocaster.yml

oc create -f crd/lespaul.yml

oc get guitars -o yaml

Running the Additional UI

To ease testing, you can also use the provided UI so you can list, create, and delete guitars.
The UI uses flask and the same client. For that, we deploy an additional pod and expose its route.

oc new-app karmab/sampleui

oc expose svc sampleui

SAMPLEUI=$(oc get route sampleui -o jsonpath='{.spec.host}{"\n"}')

Now you can access the URL defined as $SAMPLEUI and watch existing guitars and their corresponding review.
You can also create and delete additional guitars through the UI.

Running Your Own Code

To build your own controller using the same library, you can build your container on top of my karmab/client-python-kubernetes docker image

Conclusion

Custom controllers give you the ability to interact easily upon creation/deletion/update of objects in your cluster.
Combined with Custom Resource Definitions, they provide a nice abstraction to etcd and ease the design of your application (did someone say microservices?) within the container platform.
If you felt like all of this required learning a bunch of golang, you now know you have a possible alternative to get started. So "go" for it! (Or Python it!)