Deploying a User Provisioned Infrastructure environment for OpenShift 4.1 on vSphere

Deploying a UPI environment for OpenShift 4.1 on vSphere

NOTE: This process is not supported by Red Hat and is provided “as-is”.

This process will utilize terraform to deploy the infrastructure used by the OpenShift Installer.

Please install the appropriate version for your operating system:

This article was created utilizing Terraform v0.11.12.


Download and import the Red Hat CoreOS image for vSphere

Download the latest version of govc and use it to import the OVA:

$ cd /tmp/
$ curl -L | gunzip > /usr/local/bin/govc
$ curl -O
$ chmod +x /usr/local/bin/govc
$ export GOVC_URL=''
$ export GOVC_USERNAME='administrator@vsphere.local'
$ export GOVC_PASSWORD='password'
$ export GOVC_NETWORK='VM Network'
$ export GOVC_DATASTORE='vmware-datastore'
$ export GOVC_INSECURE=1 # If the host above uses a self-signed cert
$ govc import.ova -name=rhcos-latest ./rhcos-410.8.20190516.0-vmware.ova

This will provide a working Red Hat CoreOS image to use for deploying the user provided infrastructure environment (UPI).


Create the applicable DNS records for the OpenShift deployment

Take note of the SRV records for etcd. Also, examine the bootstrap-0 DNS records: host, api and api-int.

* A 10.x.y.38
* A 10.x.y.39
* A 10.x.y.40
_etcd-server-ssl._tcp SRV 0 10 2380 etcd-0
_etcd-server-ssl._tcp SRV 0 10 2380 etcd-1
_etcd-server-ssl._tcp SRV 0 10 2380 etcd-2
bootstrap-0 A 10.x.y.34
control-plane-0 A 10.x.y.35
control-plane-1 A 10.x.y.36
control-plane-2 A 10.x.y.37
api A 10.x.y.34
api A 10.x.y.35
api A 10.x.y.36
api A 10.x.y.37
api-int A 10.x.y.34
api-int A 10.x.y.35
api-int A 10.x.y.36
api-int A 10.x.y.37
etcd-0 A 10.x.y.35
etcd-1 A 10.x.y.36
etcd-2 A 10.x.y.37
compute-0 A 10.x.y.38
compute-1 A 10.x.y.39
compute-2 A 10.x.y.40

To learn more about required DNS the OpenShift docs discuss further.

Create a working install-config file for the installer to consume

Download the latest 4.1 installer and client from the repository.

$ cd /tmp/
$ curl -O https://
$ curl -O
$ tar xzvf openshift-install-linux-4.1.0-rc.5.tar.gz
$ tar xzvf openshift-client-linux-4.1.0-rc.5.tar.gz
$ mv oc kubectl openshift-install /usr/local/bin/
$ mkdir ~/vsphere/
$ vi ~/vsphere/install-config.yaml

apiVersion: v1beta4
  name: upi
  machineCIDR: "10.x.y.0/23"
    username: "administrator@vsphere.local"
    password: "password"
    datacenter: dc
    defaultDatastore: vmware-datastore
pullsecret: ‘omitted’
sshKey: ‘omitted’

$ openshift-install --dir ~/vsphere create ignition-configs

This will generate ignition configuration files that will need to be added to the terraform tfvars in the next section. Note, that the bootstrap ignition config is too large to be added via a virtual appliance property and will need to be stored somewhere external. The example listed below leverages a github gist.

Download the UPI terraform included in the OpenShift installer repository and update provisioning variables

Edit terraform.tfvars with the appropriate variables including the ignition configs from the previous steps:

$ git clone -b release-4.1
$ vi openshift/installer/upi/vsphere/terraform.tfvars

bootstrap_ip = "10.x.y.34"
control_plane_ips = ["10.x.y.35","10.x.y.36","10.x.y.37"]
compute_ips = ["10.x.y.38","10.x.y.39","10.x.y.40"]
machine_cidr = "10.x.y.0/23"
cluster_id = "upi"
cluster_domain = ""
base_domain = ""
vsphere_server = ""
vsphere_user = "administrator@vsphere.local"
vsphere_password = "password"
vsphere_cluster = "cluster"
vsphere_datacenter = "dc"
vsphere_datastore = "vmware-datastore”
vm_template = "rhcos-latest"

// Due to size limitations for the OVA use a URL for the bootstrap ignition config file
bootstrap_ignition_url = ""

// Ignition config for the control plane machines. Copy the contents of the master.ign generated by the installer.
control_plane_ignition = <<END_OF_MASTER_IGNITION

// Ignition config for the compute machines. Copy the contents of the worker.ign generated by the installer.
compute_ignition = <<END_OF_WORKER_IGNITION

Edit the to update the gateway and DNS server to use for VM deployment:

$ vi openshift/installer/upi/vsphere/machine/

gw = "10.x.y.254"

$ vi openshift/installer/upi/vsphere/machine/


Remove the route53 section since DNS has been created above.

$ vi openshift/installer/upi/vsphere/

module "dns" {
source = "./route53"

base_domain = "${var.base_domain}"
cluster_domain = "${var.cluster_domain}"
bootstrap_count = "${var.bootstrap_complete ? 0 : 1}"
bootstrap_ips = ["${module.bootstrap.ip_addresses}"]
control_plane_count = "${var.control_plane_count}"
control_plane_ips = ["${module.control_plane.ip_addresses}"]
compute_count = "${var.compute_count}"
compute_ips = ["${module.compute.ip_addresses}"]

Run terraform to initialize and provision the environment

$ cd openshift/installer/upi/vsphere/
$ terraform init

Initializing modules...
- module.folder
Getting source "./folder"
- module.resource_pool
- module.bootstrap
- module.control_plane
- module.compute

Initializing provider plugins...

$ terraform apply -auto-approve

module.control_plane.vsphere_virtual_machine.vm[0]: Creation complete after 40s (ID: 4203f331-984c-8103-83a9-20d760fbf605)
module.bootstrap.vsphere_virtual_machine.vm: Creation complete after 40s (ID: 4203d5ab-d5e1-4ca7-eece-66aeb4a52b7d)

Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

The environment should now be deployed in vCenter:

Inform the installer that the bootstrap process has completed and delete the bootstrap VM

$ openshift-install --dir ~/vsphere wait-for bootstrap-complete

INFO Waiting up to 30m0s for the Kubernetes API at
INFO API v1.14.0+ef1f86e up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources

$ terraform apply -auto-approve -var 'bootstrap_complete=true'

module.bootstrap.vsphere_virtual_machine.vm: Destruction complete after 7s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

The Cluster Image Registry does not pick a storage backend for vSphere platform. Therefore, the cluster operator will be stuck in progressing because it is waiting for an administrator to configure a storage backend for the image-registry. See the following bugzilla for details.

Set EmptyDir for non-production clusters by issuing the following commands:

$ cp ~/vsphere/auth/kubeconfig ~/.kube/config
$ oc patch cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'

Delete the bootstrap's DNS records and inform the installer to proceed

Note, the remaining api and api-int records should continue pointing towards the control-plane servers.

$ openshift-install --dir ~/vsphere wait-for install-complete

INFO Waiting up to 30m0s for the cluster at to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/go/src/'
INFO Access the OpenShift web-console here:
INFO Login to the console with user: kubeadmin, password: my-kube-password

Enjoy the OpenShift Container Platform 4.1 cluster on a VMware software-defined datacenter!


Cleaning up

To remove the virtual machines from vCenter, terraform can be run with the destroy option:

$ terraform destroy -auto-approve

OpenShift Container Platform, OpenShift Ecosystem