What’s New in OpenShift 3.6 – Cluster Management

This is now our second year releasing OpenShift on the Kubernetes architecture.  With over 5,000 commits from 275 authors on Kubernetes 1.6, Kubernetes is the place where the open source community gathers to produce hybrid cloud solutions that allow for the largest choice in application architecture.  Red Hat has participated in 7 Kubernetes releases, 7 OpenShift releases, and over 400 various CVE, bug, and general enhancements across z-stream updates that occur on average every 3 weeks or whenever underlying software layers need updating.  We do not just provide you with Kubernetes, we offer fixes for the operating system, the linux container runtime, and the application frameworks and languages we provide out of the box.  When you add in the fact that Red Hat doesn’t just build this software, but we also use it for OpenShift Online Starter/Pro, OpenShift.io,  and OpenShift Dedicated solutions…well, it’s going to be difficult to find another company so deeply invested in container runtime and orchestration technologies with a developer user experience.  Speaking of OpenShift Online Starter, I wanted to thank the users on us-east that received early access to OpenShift 3.6!  You will notice us releasing pre-released versions of features to OpenShift Starter more frequently.  This is really a testament to our agile methodology and our quest to deliver you software drops that have already seen some action.

That is a pretty impressive back drop for today’s release of OpenShift 3.6.  Every releases has its purpose in life.  OpenShift 3.6 comes during a cycle that is responding to an influx in massive production usage of OpenShift across a variety of industry sectors.   Whether you have read our customer success stories, visited a popular Commons gathering near you, or attended this year’s Summit gathering you should be able to find example usage of the product that resonates with what you are trying to accomplish.  With that perspective, this release of OpenShift cluster management tackles the following four areas:

  • Security
  • Scheduling
  • Containerization
  • Storage



Security is never one thing.  In fact, we like to sum it up into 10 main categories.  Within those categories, OpenShift 3.6 adds TLS certificate expiration control for all cluster components, GPG image signing support, and the encryption of secrets at rest in etcd.  Note that encryption of secrets at rest in etcd is releasing later this month in OpenShift 3.6.z, so I’ll save that topic for new blog.  Let’s take a closer look at TLS certificate expiration control and GPG signed images.

TLS certificate expiration control

Rest assured that all the framework components that make up OpenShift communicate with each other over TLS encrypted TCP traffic.  The encryption is governed by a handful of certificates that are generated at the time of installation.  People have asked us for a way to change the expiry dates for these certificates, but granularly so they can allow different components to expire at different times.  OpenShift now offers a new ansible playbook for the installer called redeploy-openshift-ca that can be leveraged to change the entire cluster with a single command.

$ ansible-playbook -i inventory openshift-ansible/playbook/byo/openshift-cluster/redeploy-openshift-ca.yml

The playbook offers the following segmentation of components.

#Certificate Authority, Node, and Master certificate expiry

#Registry certificate expiry

#Etcd, peer, server and client certificate expiry


GPG Signed Images

For a few releases now, the RHEL operating system has allowed you to GPG sign container images.  GPG is the same signing technology many infrastructures leverage to sign RPMs and so it is familiar to many people that need to put controls in place to validate where artifacts come from, that they have not changed, and declare where they are allowed to be deployed.  OpenShift 3.6 offers platform administrators a new security layer to further control what content is allowed to run inside the cluster, while at the same time, offering users an ability to determine if the image they have been given is secured.

After generating GPG keys, the platform administrator can establish a separation of duties and assign anyone the role of image-signer.  This new role can be given to continuous delivery solutions, admins, etc.  That role now has the ability to sign images and store/retrieve their signature in the OpenShift container registry.  They then take the related keys and deploy them onto specific RHEL hosts in the OpenShift cluster that are allowed to run those signed images.  Different images can have different signatures and their relationship to the nodes can be maintained by the platform administrator.



If only DMZ signed images are allowed to start on a DMZ node, you can leverage the GPG signing feature to insure that happens.  If only the UAT test team is allowed to promote a unit build to the QE end-to-end team, you can enforce that by allowing them the image-signer role and deploy the resulting required key to the end-to-end nodes that have been connected to their project via a nodeSelector.

Users now have updated metadata about the signed status of the images on the cluster.

$ oc describe istag strangeImage:latest
Image Signatures:
Name: sha256:c13060b74c0348577cbe07dedcdb698f7d893ea6f74847154e5ef3c8c9369b2c@f66d720cfaced1b33e8141a844e793be
Type: atomic
Status: Unverified


$ oc describe istag myImage:latest
Image Signatures:
Name: sha256:c13060b74c0348577cbe07dedcdb698f7d893ea6f74847154e5ef3c8c9369b2c@f66d720cfaced1b33e8141a844e793be
Type: atomic
Status: Verified
Issued By: 172B61E538AAC0EE
Signature is Trusted (verified by user "system:admin" on 2017-04-28 12:32:25 +0200 CEST)
Signature is ForImage ( on 2017-04-28 12:32:25 +0200 CEST)



Distributed systems rely on an intelligent scheduler to get the most out of your resource investments.  There are too many new features related to the scheduler to mention in detail.  I would encourage you to check out the release notes for more information.  etcd moved to a new v3 data model that enhances the overall performance of the cluster.  The cluster can now support up to 2,000 nodes.  Taints and Tolerations can assist in allowing users to target specific nodes with a higher level of logic.  Affinity and anti-affinity is now available at both the node and pod level.  And platform administrators now have a more accurate way to determine cluster capacity at any given time.  Let’s briefly take a deeper look at the new features related to placement logic and cluster capacity.

Taints and Tolerations

Allows you to taint a node so that pods without a toleration for that same taint can’t use the node. Taints are placed on the nodes (and therefore resources held by those nodes) and tolerations are placed on the pods (really workloads looking for resources) on the cluster.  Think of licensed software you might want to run on the cluster.  Before taints and tolerations you would have leveraged a nodeSelector to connect a project or image to a specific node or group of nodes.  That worked great, but it did not keep the rest of the cluster from sending workloads to those same nodes if those nodes had free resources.  With taints and toleration, you can taint the license software nodes from running anything but licensed software regardless of which projects deploy it.

Affinity and Anti-Affinity

Placement logic should never force you to hardcode selection criteria.  The more you need to limit your deployment, the less you are going to take advantage of your cloud’s resources.  Let us say you want to allow kube the freedom to select which zone an application lands in, but whichever it chooses you’d like to make sure another component of that app lands in the same zone. Or let’s say you have two app components that, due to security reasons, cannot be on the same physical box. But you don’t want to lock them into labels on nodes.


You have an ability to state these affinity and anti-affinity at the node level or at the pod level.  The feature allows for a wide range of compare operators (In, NotIn, Exists, DoesNotExist, Lt) while allowing you to declare if the matching logic is required or just preferred.  Preferred means to try, but still deploy if you can’t find the environment to match the affinity logic.


Cluster Capacity

Just like a disk drive, a cluster can become fragmented over time.  When you ask the cluster how much space is left, the addition of all the free space doesn’t indicate how many actual workloads of a specific size can run.  For example, it might say there is 10gb left, but it could be that no single node has more than 512mb available.  To solve that problem, we created a new container that you can launch as a cmdline or a Job on the cluster. The container allows you to deploy a popular workload (image) on your cluster with a commonly requested CPU/MEM limit and request. The logs from the container will tell you how many of that workload can be deployed.


oc log output:

small-pod pod requirements:
- CPU: 150m
- Memory: 100Mi

The cluster can schedule 52 instance(s) of the pod small-pod.

Termination reason: Unschedulable: No nodes are available that match all of the following predicates:: Insufficient cpu (2).

Pod distribution among nodes:
- 26 instance(s)
- 26 instance(s)


There are some really awesome features related to containers that I want to draw your attention to in OpenShift 3.6.  The kublet daemon that interacts with the container runtime has switched to leveraging the Container Runtime Interface (CRI) that is fully compatible with the Open Container Initiative (OCI) thus future proofing your container platform for years to come.

Users can now auto-prune their images back to certain numbers based on whether or not the container deployment was successful or a failure.

We introduced two new ways to get environmental variables into your containers.  When you mount a memory backed volume into a container, it leverages a directory. Now you can place (via the “projected” API in the volumes, configMap, or downwardAPI definition) all sources of configuration for your application (supplied by configMaps, secrets, and downward API) into the same directory path without worrying about file system path (directories) collisions.  Plus, you can turn (via the envFrom API in the pod spec) configMaps into environmental variables.  Instead of individually declaring environment variables in a pod definition, a configMap can be “imported” and all of its content can be dynamically turned into environment variables for the container.

Init containers provide a chained, pre-deployment mechanism for running arbitrary containers/programs as part of getting ready for your application to run. They aid in application deployment (pre-check, pre-start coordination, data population, etc) and, to some degree, dependency management.  They run to completion and each container must finish before the next one starts, and will run for each pod instance. The init containers will honor the restart policy.  Leverage initContainers in the pod spec.

Users love OpenShift deploymentConfigs, image triggers, and imageStream.  But sometimes using those abilities means you can’t use various Kubernetes API resources at the same time such as Jobs, Statefulsets, etc.  We have released a tech preview ability to combine the best of both worlds.   Now you can leverage what you like across a variety of use cases.  Imagine using an imageStream from a batch job!


This would not be a new OpenShift release without giving you more features for application persistence.  Lots of features that range from iSCSI CHAP support, controllable mount options, increased quota control, Amazon ElasticFileSystem and VMware vSphere dynamic provisioners, plus a better install experience for Container Native Storage (CNS).  Let’s take a closer look at two of them.

StorageClass Quota Control

You will recall in the last 2 releases we delivered dynamic persistent storage.  That feature exposed you to StorageClasses for the first time.  Now in OpenShift 3.6 you can control what classes of storage projects are allowed to access, how much (total size) of that class, as well as how many claims.  This feature leverages the ResourceQuota object and allows you to call out storage classes by name for size and claim settings.


$ oc create quota my-quota-1 --hard=slow.storageclass.storage.k8s.io/requests.storage=20Gi,slow.storageclass.storage.k8s.io/persistentvolumeclaims=15

$ oc describe quota my-quota-1

Name:                                      my-quota-1
Namespace:                                 default
Resource                                             Used   Hard
--------                                             ---- -----       slow.storageclass.storage.k8s.io/persistentvolumeclaims 0 15

slow.storageclass.storage.k8s.io/requests.storage            0 20Gi

Controlling who has the right to consume your most expensive storage options while also declaring how much size they are able to provision, is an obvious use case for this feature.  This feature can also be used to make sure a specific storage volume (maybe one with data only a specific project is allowed to see) can only be asked for and received by the project with the quota access to the StorageClass describing it.  This is a very versatile feature!

Container Native Storage (CNS) Install Automation

Have you ever just wanted to snap your fingers and magically conjure up persistent storage for OpenShift that happened to be a scalable network filesystem built from off the shelf servers so you could create large distributed storage solutions for media streaming, data analysis, and bandwidth intensive tasks?  OpenShift 3.6 delivers an incredible installation experience for CNS.  By simply telling us 3 nodes in the environment to use for disks, we will do the rest.  Literally all the way from setting up the OpenShift container registry to leverage the resulting glusterFS to carving out some space and enabling the CNS dynamic storage provisioner for application storage.




If you have been sitting on the side line watching the evolution of containers, orchestration, distributed systems, declarative systems, and hybrid cloud solutions I think OpenShift 3.6 is for you.  It has the cutting edge innovation you would expect, without removing the production ready qualities you require.  OpenShift places no expectation on your application while offering a healthy dose of enabling content and patterns for cloud native architectures.  Be sure to check out the other excellent OpenShift 3.6 blogs on https://blog.openshift.com.  If you have a browser, you can experience OpenShift at OpenShift Online.  If you have a laptop, OpenShift is as easy as typing oc cluster up.  Check it out today!



News, OpenShift Container Platform
, , ,