Considerations on OpenShift PKIs and Certificates

OpenShift features several Public Key Infrastructures (PKIs) that manage certificates for specific purposes. To help deploy OpenShift more securely, it’s necessary to know what each of these infrastructures does and how to best configure them.

Note that the information discussed in this article refers to OpenShift 3.x and it is subject to change in the 4.x line.

What is a PKI?

A PKI is a hierarchy of certificate authorities and certificates, similar to a tree (there are more components in a PKI, but for the sake of this discussion we are going to focus on the certificate hierarchy). The root is the Root CA. The Root CA is always a self-signed certificate.

The branches are subsidiary (sometimes also called intermediary) certificate authorities.

The leaves are the certificates that systems, users and services can directly use to identify themselves.

Leaf certificates (also known as end-entity certificates) are signed by the branches to which they are attached. Each branch is signed by a higher order branch until the path gets back to the Root CA, which, as mentioned previously, is self-signed.

For a complete PKI, together with this hierarchy, there are the processes and infrastructure around it to operate the whole system. In certain cases, the Root CAs are stored in secure disconnected locations and only SubCAs are used to create certificates. A PKI also involves processes to request and produce new certificates. The processes used to be mostly manual (managed by a dedicated team), but in recent times automation has started to be adopted also in this area.

Notice that the primary purpose of a PKI is to provide the ability to authenticate systems (mostly the server’s side of the communication). If all one needed was an encrypted connection,  self-signed certificates could be utilized everywhere (of course encryption only is not enough, so we need CA-signed certificates).

With an understanding of PKIs, let’s review the various ways they are utilized in OpenShift.

OpenShift Platform PKI

The platform PKI is dedicated to managing the platform components’ certificates. This includes: the master API, controllers, and nodes. Also, by default, most of the infrastructure components deployed at install time are signed by this CA; for example: metrics, logging, and the default router wildcard certificate. Each of these defaults can be overridden with the correct Ansible inventory parameters, if desired.

At install time, it is possible to let the installer create a new PKI, or to pass a SubCA and let OpenShift make use of it. In this case, OpenShift has full control of that SubCA including access to its private keys.

By default, the PKI CA related files are stored on all OpenShift masters in the /etc/origin/master directory as shown below:

[root@master-0 ~]# ls -la /etc/origin/master/ca*
-rw-r--r--. 1 root root 1070 Nov 15 11:06 /etc/origin/master/ca-bundle.crt
-rw-r--r--. 1 root root 1070 Nov 15 11:05 /etc/origin/master/ca.crt
-rw-------. 1 root root 1679 Nov 15 11:05 /etc/origin/master/ca.key
-rw-r--r--. 1 root root    3 Nov 15 11:20 /etc/origin/master/ca.serial.txt

Supplying a SubCA at install time can be used, for example, when a company has an existing PKI and a new SubCA is generated for use with OpenShift.

Etcd PKI

Etcd cluster has its own PKI. The PKI Root CA files can be found in the /etc/etcd/ca directory:

[root@master-0 ~]# ls -la /etc/etcd/ca
total 48
-rw-r--r--. 3 root root  1895 Nov 15 10:58 ca.crt
-rw-r--r--. 1 root root  3272 Nov 15 10:58 ca.key

The ectd PKI cannot be customized at install time. It is also very complicated to replace the Root CA and the associated certificates that have been generated with it at a later time. Therefore, it is recommended to use the Root CA for the ectd PKI.

The redeploy-ca.yml playbook can be used to re-deploy a new CA (for example in case there is suspicion of a compromise).

Aggregator Proxy PKI

This PKI is dedicated to managing certificates for extension api servers. This feature allows an extension of the control plane API by adding services dedicated to handling specific APIs (it is an alternative way of extending the API to Custom Resource Definitions (CRDs) and controllers).

There is no support to customize this PKI. The files of the Root CA can be found in the OpenShift configuration directory as shown below:

[root@master-0 ~]# ls -la /etc/origin/master/frontproxy*
-rw-r--r--. 1 root root 1078 Nov 15 11:06 /etc/origin/master/frontproxy-ca.crt
-rw-------. 1 root root 1675 Nov 15 11:06 /etc/origin/master/frontproxy-ca.key
-rw-r--r--. 1 root root    3 Nov 15 11:06 /etc/origin/master/frontproxy-ca.serial.txt
encrypted routes.

Application-level PKI

OpenShift also features an application level PKI. This PKI manages certificates generated with the service serving certificate secret feature. The Root CA files for this PKI can be found as shown below:

[root@master-0 ~]# ls -la /etc/origin/master/service-*
-rw-r--r--. 1 root root 1115 Nov 15 11:06 /etc/origin/master/service-signer.crt
-rw-------. 1 root root 1675 Nov 15 11:06 /etc/origin/master/service-signer.key

This feature allows OpenShift the ability to generate certificates that can be used for pod-to-pod communications because they present the service name in the Subject Alternative Name field (SAN).

The application level PKI cannot be initialized at install time with a SubCA. However, it is possible to replace the Root CA with a SubCA after the installation.

Security Considerations on OpenShift-managed PKIs

As we saw from the previous list, Openshift-managed PKI can be installed with a RootCA or an externally supplied SubCA (some of these PKIs are more difficult to configure this way than others).

Recommendation: use the self-generated PKI for OpenShift as opposed to a SubCA as it diminishes the impact of the CA private key being compromised.

To understand why, we need to review how certificates are validated and the risks associated when the CA private key is compromised.

Certificate validation

There are several steps in the certificate validation algorithm, but the one area of concern we need to focus on is the validation of the chain of trust.

In order to validate that a certificate has been issued by a trusted Certificate Authority, the validator needs to be able to walk back the chain of trust from the presented certificate through all the intermediary CAs up to the Root CA. Here is an example of how the chain of trust works

Source Wikipedia

In order for this to work successfully, all of the CAs public keys (both the intermediary one as well as the root one) must be available to the client.

Normally a client, such as a browser, will only hold (and therefore trust) the Root CA of any trusted PKIs. However, during the TLS handshake, the server is allowed to send the entire CA certification path together with the certificate to be validated.

So, in order for a client to trust a certificate, the only item that it needs to trust is the Root CA, regardless of the depth of the chain of trust.

What Happens When the OpenShift PKI is Compromised

What is the risk when a PKI is compromised? The risk is that a hacker may forge new certificates that can pretend to be a legitimate server on any domain and these certificates will be trusted by the rest of the system.

If the self-generated PKI was used for OpenShift, the impact of it being compromised is contained to the OpenShift cluster. To remediate this situation, one can either redeploy all the certificates (by running the redeploy-openshift-ca playbook) or simply destroy and recreate the cluster.

If a SubCA was used to install OpenShift, the impact extends to any system that trusts the RootCA which that SubCA was directly or indirectly generated from. In a company where the internal PKI was used to create a SubCA for OpenShift, this can impact the entire internal network (both client and server machines). In fact, any client that trusts the company root CA would also trust a certificate forged with the stolen SubCA.

If a company’s SubCA is utilized, impact can be mitigated in the following ways:

  1. Use the name constraint extension to scope down the domain of the certificates that can be generated by the SubCA. Ideally, the OCP cluster in question would be confined to its own domain, and everything else in the company would use different domains.
  2. Try to detect if there are rogue certificates circulating in the company environments by using certificate transparency logging approaches.
  3. If there is a suspicion that the OCP CA has been compromised, revoke it. This will also imply that all the certificates issued by that CA are no longer valid.
  4. Use mechanisms such as certificate revocation list and OSCP stapling to make sure that clients on the company network do not trust revoked certificates. Be aware though that, the author of this article discusses how both these mechanisms do not fully guarantee that revoked certificates will not be trusted.

In conclusion, because the impact of a compromised SubCA propagates to the entire company’s network, the recommendation is to keep the OpenShift PKIs separate from the company main PKI, which implies using a Root CA for the OpenShift PKIs.

Notice that even with the Root CA (which is self-signed by definition), there are no self-signed platform certificates in an OpenShift deployment.

Because the services using the certificates generated with the OpenShift-managed PKIs form closed systems, there is no need to establish any level of trust with the rest of the company’s network. Therefore, using a Root CA for these PKIs does not introduce any limitations in the use of OpenShift.

There is, however, a limited set of endpoints that need to be accessible and trusted from outside of OpenShift.

External endpoints

Some platform components, such the master API external endpoint, metrics, logging, registry etc… are exposed externally and it is expected that users in the company network will connect to them. For these components, one should configure company-issued certificates so they can be trusted by external users.

The table below shows each component and a link to the documentation where you can find information on how to customize the exposed certificate and install time.

Component Install-time Customization
Master external endpoint here
Metrics / hawkular here
Logging / kibana here
Registry here
Registry console N/A
Prometheus / grafana Customize the wildcard route certificate for the router.
Cluster console here
Router here


Note: this table reflects a 3.11 default setup. In the upcoming 4.x line, these configurations may change.

For those components that are not customizable at install time, you can follow the post install steps explained here. Furthermore, those components that have the certificates defined in the route, if not customized, will use the default route certificate. Each of these components can be addressed by customizing this certificate.

Wildcard certificate for the router

OpenShift, by default, makes use of a wildcard certificate in the router. When creating an encrypted route you can choose whether to use the default wildcard certificate or use a router-specific certificate.

Because router’s certificates are externally facing, they should be company-signed.

Also consider that wildcard certificates have been deprecated by the Internet Engineering Task Force (IETF which is the organization overseeing the PKI/TLS standards).

While it may be some time before wildcard certificates will stop functioning, it will mostly likely occur if this deprecation stands and we should move forward using a solution that does not rely on wildcard certificates.

Our objective should still be to allow development teams to self-provision certificates for their applications. This can be achieved by automating the certificate request process, for example using the Automated Certificate Management Environment (ACME) protocol (see for example cert-operator or cert-manager).

If your organization does not allow that capability, I recommend using wildcard certificates for non-production environments in which we expect a high number of new certificate requests Continue to use manually-provisioned certificates in production environments where fewer requests should occur and therefore the overhead of the manual requests should be acceptable.


This review of OpenShift managed certificates should help you make more informed security decisions. The good news is that in many instances, the defaults which are already in place are the best course of action. Keep in mind that, as stated in the introduction, the information presented reflects the 3.x series of OpenShift. The upcoming 4.x line of the product may change the way certificates are managed, but the security principles will remain the same.

Containers, Security
, , ,