KubeVirt – Traditional Virtualization with the New Kubernetes

Previously, in Part I of this blog series, we introduced the idea that KubeVirt is about balance:

  • Mature virtualization features and concepts, yet Kubernetes philosophy and semantics.
  • Pods and containers disappear when stopped. Virtual machines have a life cycle and their configurations persist so they can be stopped and started again.
  • Years of features and knobs for different operating system and hardware, yet keep it simple.

In this article, we’ll focus on the scope and features of KubeVirt with that balance in mind. Let’s start with a discussion of how compute resources are managed in this new world.

KubeVirt Virtualization Concepts

KubeVirt exposes a VirtualMachine entity in Kubernetes. This entity is persistent and defines the configuration of a virtual machine. This allows one to create, edit, start, stop, and start again a virtual machine (which one cannot do with a Kubernetes Pod). When the virtual machine is started, a VirtualMachineInstance is created, manifesting in Pod and Container in which the virtual machine runs.

The VirtualMachine entity allows one to define virtual machines “the way you would expect it” from a virtualization expert’s perspective. You can name them, describe the virtual hardware devices, define multiple disks and networks.

Expect to find your regular virtualization features here: CPU, memory, NUMA, CPU pinning, hugepages, CPU model selection, virtio-rng, memory overcommit, custom SMBIOS, cloud-init, boot order, serial console, graphical (VNC) console, custom PCI addresses for virtio devices, I/O threads, guest agent integration, and more being worked on.

Kubernetes Virtualization

Now let’s see what Kubernetes adds to the mix, for virtual machines too!

Simplifying with Presets

Above was a long list of features just for the compute part (there are also storage, network and management). First and foremost, we want to simplify defining a VM. No one defines every knob for every VM directly in the virtualization world, and neither do we need to with KubeVirt. Kubernetes Presets allow us to define useful sets of features together and apply them to entities. We use VitrualMachineInstancePresets to create inheritable configurations for different operating systems, compute instance types (flavors), or hardware profiles. Of course one can create their own custom Preset to suit their needs.

Secrets and ConfigMaps

Since the container image is immutable and re-used for many different instances, there is a need to pass configuration and secrets when instantiating a container. KubeVirt supports both Secrets and ConfigMaps and exposes them to the virtual machine as a disk similarly to a container, allowing us to use the same approach across both containers and virtual machines.

Scheduling – node selection, affinity, anti-affinity, taints and tolerations

Sometimes you want to run virtual machines on the same hypervisor host (affinity), on a specific host (node selection), or across separate hosts for redundancy (anti-affinity). KubeVirt virtual machines benefit from the same Kubernetes scheduling mechanism pods use.

Advanced Controllers – DaemonSets, ReplicaSets, StatefulSets, Jobs, Deployments

There are a set of controllers that provide functions related to advanced automation related to scheduling, scaling, etc.

  • DaemonSets allow us to schedule a pod per host (or set of hosts). This allows us to easily make sure a set of hosts all have a needed feature on them.
  • ReplicaSets allow us to scale out pods as needed. Changing the target number of instances required will cause the controller to scale up or down the number of instances. This is used for stateless/ephemeral workloads.
  • Deployments provide update strategies for Pods and ReplicaSets.
  • StatefulSets are similar to ReplicaSets but are designed for workloads in need of persistent storage and a clear order of operations.
  • CronJobs and Jobs allow us to schedule workloads to run until completion (“cron for pods”).

 

vmctl is one way to use these higher-level workload controllers described above with KubeVirt virtual machines. More native integrations are being explored.

Monitoring and logging

Kubernetes is usually integrated with Prometheus for metrics, and has built-in logging facilities. KubeVirt adds Prometheus compatible virtualization metrics, and writes logs to the logging facility, just like any other container. This allows the same metrics gathering and log aggregation tools to get insights into both - container and virtual machine workloads.

Future work is needed to allow processes inside the VMs to utilize the Kubernetes abstraction for logs and metrics.

Ecosystem

KubeVirt provides the same functionality one expects from a virtual machine. This allows one to easily integrate it with existing virtualization tools like Ansible, Fog, Foreman, Vagrant, and other tools are in the plan like jclouds, Packer, Terraform, Helm charts, backup solutions and more.

Virtual machines are defined declaratively through the same API just like any other entity in Kubernetes. Furthermore, KubeVirt also uses the native Kubernetes API for events, Volumes, ConfigMaps, metrics, logs and more. Thus tools which operate like Prometheus, Helm, kubectl, Loki, and kustomize (to name a few), will directly be able to interact with KubeVirt entities as well.

As an example, kubectl can be used with virtual machines, just try to wait for a VM to be ready by using kubectl wait --for condition=Ready vm myvm .

Creating and Provisioning Virtual Machines

Integrating with existing tools in the virtualization ecosystem is important as it allows users to use the tools they know in the same way. For many organizations, this means providing features like layer 2 network for DHCP, and PXE boot or cloud-init are needed in order to keep existing flows. Another aspect crucial to virtual machines is the creation of virtual machines from an existing image/template. While containers also use images, there is a crucial difference. Container images are immutable. Virtual machines need to clone the image so they can change it in a persistent manner (other than ephemeral virtual machines).

See Containerized Data Importer (CDI) for more information on cloning, importing or uploading disks.

v2v – Importing Virtual Machines

KubeVirt uses KVM, so KVM images are easy to upload. One can also use the virt-v2v utility to convert and upload VMware virtual machines. As part of enabling migration and making it easy to import/convert virtual machines, an Import VM APB has been developed, and a UI is being worked on to easily point to an existing VMware environment and import the virtual machines – configuration and disks.

Customizing with Hooks

Another aspect of keeping a balance is not exposing every potential virtualization knob. Hooks are a powerful tool allowing changes to the flow, behavior and configuration for custom needs without adding every potential use case to KubeVirt (which would add complexity). Hooks are also great to work around a missing feature until it is added to KubeVirt.

As an example, this smbios hook allows users to manipulate the libvirt XML to inject smbios information to the virtual machine.

What Is Missing in Virtualization on Kubernetes?

The main gaps to support production-ready workloads, are currently live migration, fencing, and hot-plug.

Live migration is a marquee feature in the virtualization world. It is actually one of the first features KubeVirt implemented in early versions to showcase its potential, then it was dropped in favor of re-architecting some internal implementation details. Live migration is being worked on here. The latest live migration capability was released in KubeVirt 0.12.0.

Fencing is required to handle node failures, to make sure it is safe to start the virtual machine on another node, without corrupting the disk (if the original host/virtual machine is still up and writing to it as well). This is beneficial to the broader Kubernetes ecosystem and is being worked on in collaboration to cover the various use cases.

Hot-plug allows adding and removing resources while the virtual machine is running. The common ones are CPU, memory, network and disk. Mutable compute resources are a topic under discussion in Kubernetes which would be relevant for virtualization as well, such as the Vertical Pod Autoscaler.

Stay Tuned for Storage and Networking

In this post, we have discussed some of the compute and management related features available for running virtual machines with KubeVirt and Kubernetes. Check out our Part III post coming next which will dive into storage and networking features.