With the release of Red Hat OpenShift 4, the concept of User Provisioned Infrastructure (UPI) has emerged to encompass the environments where the infrastructure (compute, network and storage resources) that hosts the OpenShift Container Platform is deployed by the user. This allows for more creative deployments, while leaving the management of the infrastructure to the user. This allows us to deploy OpenShift Container Platform 4.1 on a mix of virtual and bare metal machines.

A few words on why we are deploying on a mixed environment and not only bare metal or virtual machines. On one hand, virtual machines provide additional protection because they can be migrated live to avoid downtime during maintenance. They can also be automatically restarted in case of hypervisor failure, while bare metal machines mainly rely on backup/restore procedures. So, our most precious assets can benefit from running on virtual machines. Bare metal on the other hand provides raw power without the virtualization overhead and extra cost, and can also provide higher density. In the context of OpenShift, a hybrid deployment offers the security and high availability of virtualization for the control plane, with the power and lower cost of bare metal for the applications.

OpenShift Architecture

Let's start by describing the platform. A standard deployment of OpenShift Container Platform 4.1 will comprise the following machines:

• Bootstrap - a temporary machine that runs a minimal Kubernetes used to deploy the OpenShift control plane. It will be deleted at the end of the installation.
• Masters - the core of the control plane. They run the etcd cluster, as well as the core operators. We usually deploy three masters for high availability, because it requires a quorum.
• Infra nodes - the nodes where we run the base services, that we can also consider part of the control plane, such as the image registry. We usually deploy two infrastructure nodes for high availability.
• Compute nodes - the nodes where we run the applications. We usually deploy at least three compute nodes, in order to natively support clustered applications that require a quorum. And these nodes will be deployed on bare metal machines to allow higher density and limit the cost of virtualization.

We also use an additional virtual machine that acts as a bastion. It will also host the infrastructure services that we use to deploy OpenShift. The services are pretty simple and don't consume a lot of resources, so the bastion can be as small as 1 vCPU and 2 GB of memory.

The following table summarizes the machines that we use to deploy OpenShift:

Operating system

Red Hat Enterprise Linux 8 was recently released, so we'll honor the great engineers behind it by running everything on RHEL 8^[b]. We simply download the DVD ISO image from access.redhat.com and install the bastion from it, using the minimal server profile.

During the installation, we configured the network as follows:

• Hostname: bastion.ocp67.example.com
• IP address: 192.168.67.254
• Netmask: 255.255.255.0
• Gateway: 192.168.67.1
• DNS server: 192.168.67.1
• DNS domain: ocp67.example.com

Then, our first step is to register the machine with Red Hat Subscription Manager and enable the base repositories for Red Hat Enterprise Linux 8:

The bastion is now ready to host the infrastructure services.

DNS

The first service that we need is DNS. If you have a company-wide DNS configuration, you should consider adding the records there. For our deployment though, we provide the steps to install and configure ISC Bind as the DNS server for the ocp4.example.com zone.

First, we install the bind-chroot package:

We then need to modify the /etc/named.conf file to:

• Listen on the IP address of the bastion, i.e. 192.168.67.254
• Allow queries and recursion for the local network, i.e. 192.168.67.0/24, via an ACL
• Disable DNSSEC validation, because we don't have the full chain of trust
• Associate the forward (ocp67.example.com) and reverse (67.168.192.in-addr-arpa) zones to files

We need to create the forward zone file, /var/named/ocp67.example.com.zone, to store the zone records.

And the reverse zone file, /var/named/67.168.192.in-addr.arpa.zone, contains the PTR records.

We can now start the named-chroot service:

And allow the service to be accessed at the firewall level:

Now that our DNS server is working, we should use it on the bastion as well. To do that, we simply need to update /etc/resolv.conf.

TFTP

Depending on the machine, booting from the network will be done via PXE or iPXE. In our lab, the physical servers still rely on PXE. However, it's possible to make them use iPXE by loading a specific image via PXE. To do so, we need a TFTP server to export this image.

The first step is to install the tftp-server package, as well as the iPXE boot images. They are available in RHEL 8 repositories, so we simply use DNF:

Then, we create the standard tree for the TFTP server and link the image:

We can now start the tftp service:

And allow the service to be accessed at the firewall level:

DHCP

The next service we need is DHCP. It provides IP addresses to the machine interfaces, based on their MAC addresses, and it also provides the iPXE information. For this service, we install and configure ISC DHCP server. The package is natively available in RHEL 8, so we install it with:

Then, we create the /etc/dhcp/dhcpd.conf file with static MAC / IP mapping, as well as all the options we want the machines to receive, such as router, DNS server, domain name, etc…

Note the statements to configure the PXE / iPXE protocol:

We specify the TFTP server with next-server. Then we answer differently depending on the protocol used. For iPXE, we provide the URL to the iPXE script provided by matchbox. For PXE, we provide undionly.kpxe, that simply chainloads an iPXE call, emulating the iPXE behavior.

We can now start the dhcpd service:

And allow the service to be accessed at the firewall level:

Matchbox

According to its documentation, Matchbox is a service that matches machines to profiles to PXE boot and provision Container Linux clusters. And that's exactly what we want because OpenShift 4 is based on Red Hat Enterprise Linux CoreOS.

The installation documentation says that no package for RHEL is available due to Go version conflicts, and that we need to follow the generic deployment method, based on pre-built binaries. So, the first step is to download and extract the latest release: v0.8.0.

Then we copy the matchbox binary to /usr/local/bin, so that it's available in the PATH. We also copy the example systemd unit in /etc/systemd/system to allow starting matchbox at boot.

We need to create the standard directories that matchbox will look for:

And we populate them with the required files for bare metal:

We also create the different profiles we need to deploy OpenShift 4.1: bootstrap, master, infnod and cptnod.

/var/lib/matchbox/profiles/bootstrap.json

/var/lib/matchbox/profiles/master.json

/var/lib/matchbox/profiles/infnod.json

/var/lib/matchbox/profiles/cptnod.json

And we also create the groups files to associate the MAC addresses of the machines to the profiles, i.e. associate the machines to their role in the OpenShift 4.1 deployment.

/var/lib/matchbox/groups/bootstrap-0.json

/var/lib/matchbox/groups/master-0.json

/var/lib/matchbox/groups/master-1.json

/var/lib/matchbox/groups/master-2.json

/var/lib/matchbox/groups/infnod-0.json

/var/lib/matchbox/groups/infnod-1.json

/var/lib/matchbox/groups/cptnod-0.json

/var/lib/matchbox/groups/cptnod-1.json

/var/lib/matchbox/groups/cptnod-2.json

The matchbox binary expects to run as matchbox user, so we create it:

We can now start the matchbox service:

And allow the service to be accessed at the firewall level:

HAProxy

The OpenShift 4.1 control plane provides 2 services: API and router. The API allows the clients to interact with the cluster, and the router allows access to the application hosted on the cluster. Among these applications is the OpenShift Console, for example.

The control plane is deployed in a highly available configuration with 3 masters and 2 infrastructure nodes, so we need a load balancer to redirect the traffic to the different nodes, depending on the status. HAProxy is provided with RHEL 8, so we use it:

We now need to configure it via /etc/haproxy/haproxy.cfg.

We also need to enable an SELinux boolean to allow HAProxy to bind to non standard ports for the API frontends.

We can now start the haproxy service:

And allow the service to be accessed at the firewall level:

This configuration is enough for a lab, but in production we would use a highly available configuration with keepalived.

Deploying OpenShift Container Platform 4.1

We're now ready to deploy OpenShift 4.1. There are a few steps to follow, but it's pretty straightforward now that the base services are running.

SSH keypair

In order to follow the configuration, or simply to do maintenance actions on the different machines of our cluster, we will need to connect via SSH. The ignition process allows us to import an SSH public key in /core/.ssh/authorized_keys when we create the machine. So, we create a new SSH key pair on the bastion:

Pull secret

We also need to get our secret that will be used to pull the images needed to run OpenShift. This pull secret can be retrieved from https://cloud.redhat.com/openshift/install/metal/user-provisioned. On this page, we find a "Download Pull Secret" button that will store the file on your machine. We recommend keeping it, as we may need it for other OpenShift clusters. In the following steps, we consider that the pull secret is in /root/ocp4_pull_secret on the bastion.

Installation configuration

The OpenShift installation mainly relies on the ignition process that configures the machines with their appropriate roles. Creating the ignition files can be really painful and it is also error prone. Fortunately, there is a tool to generate these file: openshift-install. To install it, we simply download it from the OpenShift mirror:

Later, we will also need the oc command to interact with our cluster. We download it from the same mirror:

The openshift-install command requires a work directory to store the assets it generates:

The openshift-install command is able to work in interactive or unattended modes. To be able to deploy on UPI, we need to use the unattended mode and provide an answer file that will be used to generate the assets. The file has to be named install-config.yaml and exists in the work directory. Here is the file we use:

We can now use openshift-install to generate the ignition files:

The work directory now contains the ignition files that we already reference in the matchbox configuration. We simply need to copy them into the matchbox directory:

We can now boot the virtual machines in PXE mode, in no specific order. When booting the machines use the ignition configuration to install the cluster. We can monitor the deployment via openshift-install command. It checks that the initial cluster operators are available:

While the installation proceeds, we can collect the Kubernetes configuration file generated by openshift-install, at the same time as the ignition files.

This file contains the credentials to connect to the API. And we will use them to configure the storage for the image registry. During the deployment, no storage backend is specified to allow us to set what we want. In production, we would select a resilient storage solution. As this is not the main topic of this article, we will stick to ephemeral storage, simply by configuring an empty directory.

After some time, the image-registry cluster operator will be available. We can now validate that the deployment is complete. For that, we can again use openshift-install:

We now have a fully working control plane with infrastructure nodes. We can access the OpenShift console via the following link: https://console-openshift-console.apps.ocp67.example.com. To login, the username is kubeadmin and we can find the password in /root/ocp67/auth/kubeadmin-password.

We can now delete the bootstrap node, because it is only used during the initialization phase. We can also remove it from the HAProxy backend pools.

Bare metal nodes

Now that our OpenShift environment is up, we can deploy the bare metal compute nodes, as they will be used to run the applications. This is pretty straightforward. The native VLAN on the main network interface is the same as the one connected to the bastion's interface. That ensures that DHCP will work.

To make the bare metal machines boot via PXE, we need to configure them. Here, we only show the basic steps that are run as IPMI commands. Depending on the hardware vendor, there can be other steps required to perform to enable PXE on certain interfaces that are not available via IPMI commands.

To be able to send IPMI commands, we need to install ipmitool. It is part of RHEL 8.

Then, we need to tell each bare metal machine to boot over PXE during the next boot. This is done by setting the chassis bootdev to pxe. And finally, we reset the power state of the bare metal machines.

The compute nodes will boot via PXE and install the required components. A few minutes later, they will appear in the list of nodes and we will be able to deploy an application. The service catalog provides some great examples to start with.