How to use GPUs with Device Plugin in OpenShift 3.9 (Now Tech Preview!)

This post updates the previous version based on OpenShift 3.6 with relevant changes for OpenShift 3.9, namely the introduction of Device Plugins. Device Plugin support is now marked as Tech Preview in OpenShift 3.9.

Introduction

The inclusion of Machine Learning and Artificial intelligence components has quickly become the hallmark of many next generation applications.  Being able to quickly learn from vast amounts of data is critical to industries such as ecommerce, life sciences, financial services and marketing.  Within the last several years, Graphics Processing Units (GPUs) have become key to optimizing the process of learning from data. Because of the wide applicability of ML/AI, (and the impressive performance of GPUs), combining the developer-centric “source to image” workflow espoused by OpenShift with GPU hardware can increase developer productivity by allowing for fast and safe access to GPU hardware.

Providing access to GPU hardware is just the first step – once enabled, a cluster provides development teams with the ability to run machine learning frameworks, such as Caffe, TensorFlow, Torch, MXNet, Theano, and even raw access to CUDA.  With these frameworks, developers can bring next generation data-centric feature sets to their applications, and revolutionize customer experience.

This blog post will show how to use GPUs with Device Plugin in OpenShift 3.9. I start with a description of the environment, how to setup the host including installing the essential NVIDIA drivers and NVIDIA docker container runtime. Next, in part 2, I show how to install and enable the GPU Device plugin on OpenShift (running OCP 3.9) cluster. Finally, in part 3, I show how to take advantage of GPUs with training using Caffe2.

Environment Overview

  • Red Hat Enterprise Linux 7.4, Caffe2 container image
  • OpenShift Container Platform 3.9 Cluster running on AWS
  • Master node: m4.xlarge
  • Infra node: m4.xlarge
  • Compute node 1: m4.xlarge
  • Compute node 2: g3.8xlarge (Two NVIDIA Tesla M60 GPUs)

Host Preparation

NVIDIA drivers must be installed on the host as a prerequisite for using GPUs with OpenShift.  Let’s prepare the host by installing NVIDIA drivers and NVIDIA container enablement. The following procedures will make containerized GPU workloads possible in OpenShift, leveraging the new Device Plugin feature in OpenShift 3.9.

Part 1: NVIDIA Driver Installation

NVIDIA drivers are compiled from source.  The build process requires the kernel-devel package be installed.  Ensure that the version of kernel-devel installed matches the running kernel version.

# yum install kernel-devel-`uname -r`

The xorg-x11-drv-nvidia package needs requires the DKMS package. DKMS is not supported or packaged for RHEL.  Work is underway to remove the NVIDIA driver requirement on DKMS for Red Hat distributions. DKMS can be installed from the EPEL repository.

First, install the epel repository:

# rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

The newest NVIDIA drivers are located in the following repository:

# rpm -ivh https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-9.1.85-1.x86_64.rpm

Auxiliary tools and mandatory libraries are contained in the following two packages.  Let’s install them to have all necessary utilities. This will also pull nvidia-kmod package, which includes the NVIDIA kernel modules.

# yum -y install xorg-x11-drv-nvidia xorg-x11-drv-nvidia-devel

Remove the nouveau kernel module, otherwise the nvidia kernel module will not load. The installation of the NVIDIA driver package will blacklist the driver in the kernel command line (nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off ), subsequent reboots will prevent the loading of the nouveau driver.

# modprobe -r nouveau

Load the NVIDIA and the unified memory kernel modules.

# nvidia-modprobe && nvidia-modprobe -u

Let’s check if the installation and the drivers are working.  Extracting the name of the GPU can later be used to label the node in the OpenShift context.

# nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 | sed -e 's/ /-/g'
Tesla-M60

Adding the nvidia-container-runtime-hook

The version of docker shipped by Red Hat includes support for OCI runtime hooks.  Because of this, we only need to install the nvidia-container-runtime-hook package and create a hook file.  On other distributions of docker, additional steps may be necessary. See NVIDIA’s documentation for more information.

After verifying that the bare-metal NVIDIA drivers are working, the next step is to enable containerized GPU workloads. Install the libnvidia-container and nvidia-container-runtime repository.

# curl -s -L https://nvidia.github.io/nvidia-container-runtime/centos7/x86_64/nvidia-container-runtime.repo | tee /etc/yum.repos.d/nvidia-container-runtime.repo

The next step will install an OCI prestart hook, which is responsible for making all the needed NVIDIA libraries, and binaries available in the GPU container.  Environment variables in the container Dockerfile must be present for the hook to take effect.

NVIDIA_DRIVER_CAPABILITES=compute,utility

# yum -y install nvidia-container-runtime-hook

The next step is to make docker aware of the hook, the following bash script will accomplish this.

# cat <<’EOF’ >> /usr/libexec/oci/hooks.d/oci-nvidia-hook
#!/bin/bash
/usr/bin/nvidia-container-runtime-hook $1
EOF

Do not forget to make the hook executable

chmod +x /usr/libexec/oci/hooks.d/oci-nvidia-hook

To use the hook with CRI-O, create the following JSON file:

# cat <<’EOF’ >> /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json
{
   "hook": "/usr/bin/nvidia-container-runtime-hook",
   "stage": [ "prestart" ]
}
EOF

The last step is to label the installed NVIDIA files with the correct SELinux label.  Without this step, you will get a “Permission Denied” error if used inside a container.

# chcon -t container_file_t  /dev/nvidia*

Everything is now set up for running a GPU enabled container.  If a custom GPU container is built, make sure to include the following environmental variables in the Dockerfile or in the pod yaml description (See later examples how to use a pod with these environment variables):

# nvidia-container-runtime-hook triggers
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=8.0"   # depending on the driver

To test GPU availability and correct operation of driver and container enablement, try running a cuda-vector-add container:

# docker run -it --rm docker.io/mirrorgooglecontainers/cuda-vector-add:v0.1 
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

If the test passes, then the drivers, hooks and the container runtime are functioning correctly.


Part 2: Openshift 3.9 with GPU Device Plugin

While Kubernetes features are still in Alpha, they are disabled by default and placed behind a Feature Gate.  This means they must be explicitly enabled on each node.  When the Device Plugin feature graduates to Beta, it is enabled by default and the feature gate step is no longer necessary.

After successful installation of OpenShift 3.9, the first step is to add the Device Plugin Feature Gate to the /etc/origin/node/node-config.yaml on every node that has a GPU.

kind: NodeConfig
kubeletArguments:
 feature-gates:
 - DevicePlugins=true

Restart the atomic-openshift-node service so that the changes become active:

# systemctl restart atomic-openshift-node

Next, create a new project:

# oc new-project nvidia

The project is necessary for the creation of additional service accounts that will have different security context constraints depending on the pods scheduled. The nvidia-deviceplugin service account will have different responsibilities and capabilities compared to the standard one.  This means we must also create an additional security context constraint.

# oc create serviceaccount nvidia-deviceplugin

The following snippet shows the new security context constraint (SCC), which will be associated with the nvidia-deviceplugin service account. The Device Plugin creates sockets and mounts host volumes.  For security reasons, these two capabilities are disabled by default in OpenShift, so we chose to create a custom SCC, and use that when starting the pod.

# nvida-deviceplugin-scc.yaml
allowHostDirVolumePlugin: true
allowHostIPC: true
allowHostNetwork: true
allowHostPID: true
allowHostPorts: true
allowPrivilegedContainer: true
allowedCapabilities:
- '*'
allowedFlexVolumes: null
apiVersion: v1
defaultAddCapabilities:
- '*'
fsGroup:
 type: RunAsAny
groups:
- system:cluster-admins
- system:nodes
- system:masters
kind: SecurityContextConstraints
metadata:
 annotations:
   kubernetes.io/description: anyuid provides all features of the restricted SCC
     but allows users to run with any UID and any GID.
 creationTimestamp: null
 name: nvidia-deviceplugin
priority: 10
readOnlyRootFilesystem: false
requiredDropCapabilities:
runAsUser:
 type: RunAsAny
seLinuxContext:
 type: RunAsAny
seccompProfiles:
- '*'
supplementalGroups:
 type: RunAsAny
users:
- system:serviceaccount:nvidia:nvidia-deviceplugin
volumes:
- '*'

Now create the new SCC and make it available to OpenShift:

# oc create -f nvidia-deviceplugin-scc.yaml

Verify the new installed SCC:

# oc get scc | grep nvidia
 nvidia-deviceplugin true [*] RunAsAny RunAsAny RunAsAny RunAsAny 10 false [*]

To schedule the Device Plugin on the correct GPU node, label the node as follows:

# oc label node <node-with-gpu> openshift.com/gpu-accelerator=true
node "<node-with-gpu>" labeled

This label will be used in the next step.


Deploy the NVIDIA Device Plugin Daemonset

The next step is to deploy the NVIDIA Device Plugin.  Note that the NVIDIA Device Plugin (and more generally, any hardware manufacturer’s plugin) is not shipped or supported by Red Hat.

Below is an example daemonset which will use the label we created in the last step so that the plugin pods will only run where GPU hardware is available.  Here is a sample pod description that uses the NVIDIA Device Plugin and leverages the service account we just created along with a node selector.

# nvidia-device-plugin.yml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: nvidia 
spec:
 template:
   metadata:
     labels:
       name: nvidia-device-plugin-ds
   spec:
     priorityClassName: system-node-critical
     affinity:
       nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
           nodeSelectorTerms:
           - matchExpressions:
             - key: openshift.com/gpu-accelerator 
               operator: Exists
     serviceAccount: nvidia-deviceplugin
     serivceAccountName: nvidia-deviceplugin
     hostNetwork: true
     hostPID: true
     containers:
     - image: nvidia/k8s-device-plugin:1.9
       name: nvidia-device-plugin-ctr
       securityContext:
         privileged: true
       volumeMounts:
         - name: device-plugin
           mountPath: /var/lib/kubelet/device-plugins
     volumes:
       - name: device-plugin
         hostPath:
           path: /var/lib/kubelet/device-plugins

Now create the NVIDIA Device Plugin:

# oc create -f nvidia-device-plugin.yml

Lets verify the correct execution of the Device Plugin.

# oc get pods
NAME                                   READY     STATUS     RESTARTS   AGE
nvidia-device-plugin-daemonset-s9ngg   1/1       Running    0          1m

Once the pod is running, let’s have a look at the logs:

# oc logs nvidia-device-plugin-daemonset-s9ngg
2018/01/23 10:39:30 Loading NVML
2018/01/23 10:39:30 Fetching devices.
2018/01/23 10:39:30 Starting FS watcher.
2018/01/23 10:39:30 Starting OS watcher.
2018/01/23 10:39:30 Starting to serve on /var/lib/kubelet/device-plugins/nvidia.sock
2018/01/23 10:39:30 Registered device plugin with Kubelet

At this point the node itself will advertise the nvidia.com/gpu extended resource in it’s capacity:

# oc describe node ip-172-31-15-xxx.us-west-2.compute.internal|egrep ‘Capacity|Allocatable|gpu’
Capacity:
 nvidia.com/gpu:  2
Allocatable:
 nvidia.com/gpu:  2

Nodes that do not have GPUs installed will not advertise GPU capacity.

Deploy a pod that requests a GPU

Let’s run a GPU-enabled container on the cluster — we can use the cuda-vector-add image that was used in the Host Preparation step. Here is a pod description for running the cuda-vector-add image in OpenShift. Note the last line requests one NVIDIA GPU from OpenShift.  The OpenShift scheduler will see this GPU request and schedule the pod to a node that has a free GPU. Once the pod create request arrives at a node, the Kubelet will coordinate with the Device Plugin to start the pod with a GPU resource.

# cuda-vector-add.yaml
apiVersion: v1
kind: Pod
metadata:
 name: cuda-vector-add
 namespace: nvidia
spec:
 restartPolicy: OnFailure
 containers:
   - name: cuda-vector-add
     image: "docker.io/mirrorgooglecontainers/cuda-vector-add:v0.1"
     env:
       - name: NVIDIA_VISIBLE_DEVICES
         value: all
       - name: NVIDIA_DRIVER_CAPABILITIES
         value: "compute,utility"
       - name: NVIDIA_REQUIRE_CUDA
         value: "cuda>=8.0"
     resources:
       limits:
         nvidia.com/gpu: 1 # requesting 1 GPU

Create the file and start the pod:

# oc create -f cuda-vector-add.yaml

After a couple of seconds the container finishes and the logs can be examined:

# oc get pods
NAME                                   READY     STATUS           RESTARTS   AGE
cuda-vector-add                        0/1       Completed        0          3s
nvidia-device-plugin-daemonset-s9ngg   1/1       Running          0          9m

The pod completed execution.  Let’s have a look at the logs for any errors:

# oc logs cuda-vector-add
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Part 3: Multi-GPU Training with Caffe2

Now that we have a fully configured OpenShift cluster with GPU support, let’s create a more sophisticated workload on the cluster. For the next step one will use the docker.io/caffe2ai/caffe2 image for Multi-GPU training.

Here is a simple pod description for starting a Jupyter notebook.  Note that the pod is requesting two GPUs. Also, for testing and simplicity this pod will also use the host network. Beware that default access to the notebook is only from localhost. For accessing the notebook from external IPs, you must create a new Jupyter config that allows access.

# caffe2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: caffe2 
  namespace: nvidia 
spec:
  hostNetwork: true
  restartPolicy: OnFailure
  containers:
    - name: caffe2
      image: "caffe2ai/caffe2"
      command: ["jupyter"] 
      args: ["notebook", "--allow-root"]
      ports:
      - containerPort: 8888
      securityContext:
        privileged: true
      resources:
        limits:
          nvidia.com/gpu: 2 # requesting 2 GPU

Now point a browser to the node and open the following URL:

http://<node-with-gpus>:8888/notebooks/caffe2/caffe2/python/tutorials/Multi-GPU_Training.ipynb

The following view shows the successfully loaded notebook:

After the notebook is up and running, you can follow the tutorial.  The result will be a Convolutional Neural Network (ResNet-50) using the ImageNet dataset for recognizing Images. Now let’s start the training, and see how the setup utilizes the GPUs.

Performance of image recognition scales linearly with the number of GPUs in use:


 

Here is a excerpt from the tested images with accuracy and if correctly recognized or not:

 

Conclusion

This blog is meant to get you started with using GPUs on OpenShift 3.9, leveraging the new device plugin feature.  Red Hat is working hard with the Kubernetes community through the Resource Management Working Group to bring Device Plugins to Beta (in Kubernetes 1.10) and GA as soon as possible.

We’re also working with other hardware accelerator vendors (such as NVIDIA and others) to streamline the installation process, administrator- and user-experience for popular machine learning and artificial intelligence application design frameworks.

We encourage you to give this procedure a try, and give us feedback by filing issues in OpenShift Origin’s Github repo.

Categories
OpenShift Container Platform, OpenShift Ecosystem, OpenShift Origin
Tags
, , , ,