This Week in KubeVirt 4

This is the fourth weekly update from the KubeVirt team.

We are currently driven by

  • Being easier to be used on Kubernetes and OpenShift
  • Enabling people to contribute
  • Node Isolator use-case (more informations soon)

This week you can find us at:

This week we achieved to:

In addition to this, we are also working on:

Take a look at the pulse, to get an overview over all changes of this week: https://github.com/kubevirt/kubevirt/pulse

Finally you can view our open issues at https://github.com/kubevirt/kubevirt/issues

And keep track of events at our calendar https://calendar.google.com/embed?src=18pc0jur01k8f2cccvn5j04j1g@group.calendar.google.com

If you need some help or want to chat you can find us on irc://irc.freenode.net/#kubevirt

Sep. 29 2017
#kubevirt #weekly

This Week in KubeVirt 3

This is the third weekly update from the KubeVirt team.

We are currently driven by

  • Being easier to be used on Kubernetes and OpenShift
  • Enabling people to contribute
  • Node Isolator use-case (more informations soon)

This week we achieved to:

In addition to this, we are also working on:

Take a look at the pulse, to get an overview over all changes of this week: https://github.com/kubevirt/kubevirt/pulse

Finally you can view our open issues at https://github.com/kubevirt/kubevirt/issues

And keep track of events at our calendar https://calendar.google.com/embed?src=18pc0jur01k8f2cccvn5j04j1g@group.calendar.google.com

If you need some help or want to chat you can find us on irc://irc.freenode.net/#kubevirt

Sep. 22 2017
#kubevirt #weekly

This Week in KubeVirt 2

This is the second weekly update from the KubeVirt team.

We are currently driven by

  • Being easier to be used on Kubernetes and OpenShift
  • Enabling people to contribute

This week we achieved to:

In addition to this, we are also working on:

Take a look at the pulse, to get an overview over all changes of this week: https://github.com/kubevirt/kubevirt/pulse

Finally you can view our open issues at https://github.com/kubevirt/kubevirt/issues

And keep track of events at our calendar https://calendar.google.com/embed?src=18pc0jur01k8f2cccvn5j04j1g@group.calendar.google.com

If you need some help or want to chat you can find us on irc://irc.freenode.net/#kubevirt

Sep. 15 2017
#kubevirt #weekly

This Week in KubeVirt #1

This is the first weekly update from the KubeVirt team.

We are currently driven by

  • Being easier to consume on Kubernetes and OpenShift

This week we achieved to

In addition to this, we are also working on:

Take a look at the pulse, to get an overview over all changes of this week: https://github.com/kubevirt/kubevirt/pulse

Finally you can view our open issues at https://github.com/kubevirt/kubevirt/issues

If you need some help or want to chat you can find us on irc://irc.freenode.net/#kubevirt.

Sep. 8 2017
#kubevirt #weekly

That thing that makes KubeVirt a little different - or the Kubernetes virtualization API.

An unsual but persistent part in the KubeVirt development is the following question:

How does KubeVirt compare to virtlet, runv/hyper, oci-cc-runtime/ClearContainers, RancherVM, and alike?

It’s a good question. And it actually took a while to get a clear view on this.

Mini Bowls - FT388 Etruscan, FT385 Diabolo & FT387 Roman

To get this clear view, I’ll focus on two details:

  1. KubeVirt is not a container runtime, and thus not using the CRI. The VM runtime (libvirt) is provided in a pod instead.
  2. KubeVirt provides a dedicated API for VMs

Especially the first point is raised pretty often. The reason for this is - eventually - that it looks so obvious to just launch VMs instead of containers on the kubelet side - virtlet, runv, and oci-cc-runtime show that it really is an option (without going to much into the details here). Very often this is directly coupled to the question of why we ship our runtime in a pod.

The answer is simply that so far this setup “works”:

  • we can spawn VMs
  • our runtime is Kubernetes native
  • because everything lives in pods, we can be delivered to “any” Kubernetes cluster, without messing with the host or kubelet.

We can be a native Kubernetes add-on.

Btw - A nice side effect is that VMs become just a workload - and this is a good separation of concerns and responsibility. Kubernetes owns the platform, and we provide our application to run specialized (virtual machine) processes on this platform.

But even with these benefits: There are some details (i.e. how to get VMs into ressource groups of other pods) which are a little nasty to solve for us, which might be easier to solve in the CRI based approach.

And we see - we are a few lines into the post, and the discussion is going down the winding road of seeing which runtime approach is the best.

That’s how it often goes, but that’s not where I want to right now.

Retro Triangular Pattern Design

KubeVirt is not only a runtime. KubeVirt also has an explicit API for virtualization. Initially we used TPRs, then as Kubernetes extension capabilities evolved we went over to CRDs, and are now working on our own User API Server to be used with API Server aggregation. Regardless of this history - The important bit is that we have an explicit API dedicated to virtualization.

And this - in retrospective (and forward facing) - is the significant difference.

The explicit virtualization related API - which consists of new top level types for different things in the virtualization area (like VMS) - gives us the freedom to express all the functionality we need to provide a robust, solid, and stable virtualization platform.

The other approaches (except for RancherVM) have a different view here: They re-use (intentionally and with different motivations) the existing pod specification, to derive VMs from them. But this approach comes to it’s limits as soon as virtualization specific functionality (for example live migration) is needed. For those features pod’s don’t have an API, and annotations or other workarounds need to be used to “logically extend” the pod concept, to enable this missing (virtualization) functionality.

I’m not saying that the one way or the other is technically wrong. It’s just that both approaches might just be addressig different goals. I.e. oci-cc-runtime has clearly the goal of using VMs for isolation purpose only - So far it does not look as if they want to expose them as “fully fledged VMs”.

And to close the loop: The API is pretty independent of the runtime, and this tells me that in the end it is not that important how the VMs are run - with a runtime in a pod or via a CRI based runtime. The important bit is, that with KubeVirt the user is given the full controll over his VMs.

Side note - Instead of a dedicated API, there is also the option to extend the pod specification using annotations to cover the virtualization specific properties. The issue is, that this does not scale, and will bloat the pod specification.

Thus, the next time you are encountering this topic, you might want to think of KubeVirt as the Kubernetes virtualization API.

Sep. 7 2017
#kubernetes #virtualization #libvirt #virtlet #cri #ranchervm #runv #hyper #fedora #kubevirt
2 notes

KubeVirt at FrOSCon 2017

There was also a small talk at FrOSCON (Hello Sunday morning!), the recordings are now published as well.

Aug. 30 2017
#kubevirt #froscon
1 note

We had the opportunity to present KubeVirt at the regular OpenShift Commons Briefing. Even if KubeVirt does not yet run on OpenShift, we are still aiming for it - and Adam actually made good progress in the past week.

This session also included the new minikube based demo.

Aug. 30 2017
#kubevirt #openshift #fedora
1 note

GPG signed git commits and tags for fancy GitHub integration

Fancy things are a driving factor.

Thus recently GitHub drove me to add gpg signatures to (first) tags and now to commits in general.

The reasons is that those tags and commits stick out by getting a verified tag. You can see the small greenish box with “Verified” on the right hand side:

You need a couple of things to get there:

  1. Make sure to install dnf install gnupg2 on Fedora.

  2. A GPG key

  3. Associating your email with the key

  4. Uploaded GPG key in GitHub

  5. Configure (gpg-agent) and git to sign your commits

All of this is nicely documented at GitHub - kudos for this documentation.

Once the basics are setup, you can use this for signing commits and tags.

For tags I went a little step further and looked into git-evtag which is using a stronger hash and also recursing over submodules. The primary reason for this was to allow using the git tree as a primary artefact for code delivery. Which is appropriate sometimes, but not always.

And with all of this, you also get the fancy verified labels on release, as here:

Aug. 28 2017
#git #fedora #github #gnupg #gpg
2 notes

KubeVirt minikube demo

minikube is a nice tool to demo Kubernetes - running Virtual Machines with Kubevirt.

And as it’s a regular Kubernetes cluster - just with a single node - thus we can also use it to demo KubeVirt.

It was a little work, but now there is a WIP branch to move the KubeVirt demo to minikube here.

Or try it yourself in a few simple steps:

$ git clone -b withMinikube \
  https://github.com/fabiand/kubevirt-demo.git
$ cd kubevirt-demo
# See the README.md of how to setup minikube for this demo
$ ./run-demo.sh

And this is how it looks if you try it out yourself:

asciicast

Feedback is welcome!

As it’s not mentioned in the recording, the VMs are defined in yaml files like pods, for an example, take a look here.

Aug. 25 2017
#kubevirt #libvirt #fedora #centos #kubernetes #minikube
1 note
The slide deck presented at FrOSCon 2017. It is a small update on the KubeVirt front. Take a look if you want to understand the motivation, and key pillars of KubeVirt.

The slide deck presented at FrOSCon 2017. It is a small update on the KubeVirt front.

Take a look if you want to understand the motivation, and key pillars of KubeVirt.

Aug. 23 2017
#kubevirt #libvirtd #fedora
1 note

Connecting virtual machines to Kubernetes in a friendly way.

Friendly Camel

We can run them, but how can we connect them? That’s what we’ve been thinking about in KubeVirt for a while. We can run VMs (and even migrate them) on Kubernetes. But so far networking was private. The stock qemu user-space SLIRP (SLURP?) stack was used to connect the VMs to the public world. This allowed the VMs to reach out, but the world could not reach them.

The goal is to be able to speak to a KubeVirt VM, like you can also speak to other pods.

With KubeVirt we aim to be good Kubernetes citizens, which brings a few constraints, mainly:

  • Networking must work from within a Pod, as KubeVirt is deployed in Pods only
  • Networking must work with the default Kubernetes networking (CNI) - to be independent of the CNI network plugin

But because we want to rely on libvirtd for virtualization, we also have constraints from that side, mainly:

  • Networking approach must be supported by libvirtd

And these constraints have a few implications:

  • We need a a Pod NIC for each VM NIC and each VM NIC can only have one IP address
  • This further implies that each VM NIC has a pre-defined IP address - the one from the Pod NIC

Now, let’s tackle the story step by step.

A pod just has one NIC! Yes, that’s true. But there are already discussions how to change this. And besides that, CNI can already add additional NICs to arbitrary containers.

The inital state can be imagined like this:

libvirtd pod:
- eth0

Get a new NIC into a Pod

We can not assume that libvirt is on all hosts, thus KubeVirt needs to ship it in a Pod. Every VM we run, will be launched in the network namespace of the libvirt Pod.

Thus: For each virtual NIC which is used by any VM managed by libvirt, we need to create a dedicated NIC within the Pod.

How do we do this? We work with the assumption that CNI is on the host for now. We work with this temporary assumption, because we assume that there are ways through the Kubernetes API in future to attach multiple NICs to pods.

So, how do we do it? Generaly speaking we just call the CNI plugn on the host and ask it to create another NIC inside our container, or rather, inside the network namespace of our container.

One issue is that currently all NICs connected to a Pod are on the same IP network, this causes routing confusion, to solve this we are removing the routes which are related to this new Pod NIC.

The new state is now:

libvirtd pod:
- eth0
- eth42 (mac42)

Represent the Pod NIC in libvirt

Now that we’ve got the new NIC inside the pod, to be used with a virtual NIC of a VM, we need to find a way to represent this NIC in a way how libvirt can consume it. We could use the direct attachment type, where a macvtap is directly attached to the NIC. This would work, as long as we don’t need to speak to the host - except for the fact that we’ve got no way to provide the pre-defined IP to the VM. To achieve this we need to rely on DHCP. And the only way to provide an IP to a VM in libvirt is to create a network, with a DHCP server which will distribute the IP to a VM.

Thus we create a libvirt network for the Pod, including an entry which is mapping the mac-address of the Pod NIC to the IP of the Pod NIC. So - There is an interesting fact here: We map the Pod mac-address to the Pod IP. This means that the VM later on will have a NIC which has the same mac and IP as the Pod NIC had. This is necessary to ensure that all packets get to the VM. But to allow all packets to travel to the VM, we need to ensure that the mac and IP is not used along the path. This means we need to remove the IP from the Pod NIC, and also change the mac of the Pod NIC - because if they keep the mac and IP, then packets, addressed to the VM, would end at the Pod NIC (because mac/ip match).

Now - What did we do? We removed the IP from and changed the mac of the Pod NIC, then we created a libvirt network, which will distill into a bridge, connected to the Pod NIC, and also a dnsmasq instance providing all DNS, and IP informations to the VM once it comes up. The delivered IP will be the former IP of the Pod NIC.

After starting the network, the new state is:

libvirtd pod:
- eth0
- eth42
  - br-eth42
    # BR: DHCP and host-to-guests

Connect the VM to the network

Connecting the VM is now trivial - We use the regular libvirt network source to connect the VM to the previously defined network. Important is just to assign the correct mac address to the VM NIC. As a reminder, the mac address used to be the mac address of the Pod NIC, but now used for the VM.

With this we effectively mived the IP endpoint from the Pod NIC to the virtual NIC of the VM.

After VM boot the new state is:

libvirtd pod:
- eth0
- eth42
  - br-eth42
    - macvtap-42 (mac42)

Friendly Street

What happens on VM boot?

Once the VM boots a few things happen (okay - some of these things happen when you start the libvirt network).

A bridge will be created by libvirt, attaching the Pod NIC as a slave. A dnsmasq instance will be spawned providing DHCP to the VM. A VM will be instanciated, connected via a tun device to the bridge.

Once the VM has booted up, it can retrieve it’s IP using DHCP. This will also include the correct DNS infromations the Pod is using.

Once a ping is sent from the VM, it will travel through the TUN device, to the bridge from there on through it’s slave, the Pod NIC, to the CNI network. The CNI network will now see packets from this NIC - and it does not see that they come from a VM.

There is thus no difference to how the pod traffic looks like - so how could it be irritatted? it isn’t, that’s nice.

Note: Rancher VM has a similar approach, except that it does not use libvirt, and that it replaces the original Pod NIC, and does not additional ones.

Code? Just POC.

Jun. 10 2017
#kubernetes #libvirt #kubevirt #network #rancher
1 note

flatpak-ing Servo Nightly

Servo - that rendering engine written in Rust - can be built from source. But there are also nightly builds available.

Servo logo?

But the issue with those nightly builds is that they depend on $very old crypto libraries - which might be because the builder distro is so old - and those are not shipped on Fedora 25. This effectively makes it annoying to run these builds on Fedora.

Flatpak logo

One way to solve this issue is to provide a flatpak for these builds. Flatpaks are containers (shipping and runtime) and will thus also contain the dependencies - liek these nagging crypto libs.

However long story short, I’ve created this small repo which can be used to wrap the Servo nightly binaries in a flatpak wrapper - and it’s also adding the two annoying dependencies.

Thus - once they get published you can run Servo on your Fedora (or other flatpak distro) without issues.

Feel free to try it out:

$ git clone https://github.com/fabiand/flatpak-servo.git
$ cd flatpak-servo
$ pkcon install flatpak
# NOTE flathub seems to be down right now
$ make
$ flatpak run org.servo.Servo wikipedia.org
# OR - for browserhtml
$ flatpak run org.servo.Servo

You should be greeted with something like

Screenshot

I’m actually not sure why the browserhtml view is so broken …

Next step would be building servo from source using flatpak-builder.

It was actually pretty nice to work with flatpak.

Jun. 3 2017
#servo #rust #fedora #flatpak

Migrating VMs on Kubernetes using KubeVirt

Meh - Migrating a VM - in Kubernetes? WTH.

Kubernetes is hosting stateless pods. That’s okay. Kubevirt is running pet VMs on these stateless pods, andyou can read more about it here.

However, whenever you encounter a situation where you end up running a VM in KubeVirt, then you can now also use VM migration to migrate VMs from one host to another - using the KubeVirt API:

To level expectations, this is early work. And it surely needs to mature.

Mar. 14 2017
#kubernetes #kubervirt #fedora #virtualization #migration #centos
1 note

Booting a VM of an iSCSI LUN - inside a Kubernetes cluster

Create some entities in Kubernetes:

# Create the pod, services, persistent volumes, and claims
$ kubectl create -f iscsi-demo-target-tgtd.yaml
persistentvolumeclaim "disk-custom" created
persistentvolumeclaim "disk-alpine" created
persistentvolumeclaim "disk-cirros" created
persistentvolume "iscsi-disk-custom" created
persistentvolume "iscsi-disk-alpine" created
persistentvolume "iscsi-disk-cirros" created
service "iscsi-demo-target" created
pod "iscsi-demo-target-tgtd" created

Now use a pod to access the created target:

# Run a qemu instance to see if the target can be used
# Note: This is not testing the PV or PVC, just the service and     target
# Use ctrl-a c quit to quit
$ kubectl run --rm -it qemu-test --image=kubevirt/libvirtd -- \
  qemu-system-x86_64 \
    -snapshot \
    -drive file=iscsi://iscsi-demo-target/iqn.2017-01.io.kubevirt:sn.42/2 \
    -nographic

And enjoy the boot:

ISOLINUX 6.04 6.04-pre1  Copyright (C) 1994-2015 H. Peter Anvin et al
boot: 


OpenRC 0.21.7.818fc79999 is starting up Linux 4.4.45-0-virtgrsec (x86_64)
…
Welcome to Alpine Linux 3.5
Kernel 4.4.45-0-virtgrsec on an x86_64 (/dev/ttyS0)

localhost login:

Okay - It’s not all end-to-end yet - when looking at it from a KubeVirt perspective.

What you see is that a qemu instance is booting of an iSCSI target LUN, offered by an iSCSI portal running as an unprivileged pod on a Kubernetes cluster.

But that’s already nice, what was achieved so far:

  • iSCSI target pod with demo content
  • Service, volumes, claims to expose the content
  • qemu can boot from the LUN
  • VM object can be created to boot of the LUN (if you specify the LUN)

The remaining gap (which is being worked on) is to allow specifying a claim in the VM object (instead of the LUN as we did it above). And then we can use KubeVirt to boot a VM of a disk image - probably.

Feb. 28 2017
#kubernetes #kubevirt #qemu #libvirtd #fedora #iscsi

Containerized iSCSI Demo target

If you want to boot a VM of a disk, then you obviously need a disk.

And as I’m currently playing with launching VMs of iSCSI target’s I took this as an excercise to come up with a nice setup to easily deploy iSCSI targets on a Kubernetes cluster to provide demo images for booting VMs.

Initially I went with writing up a small docker image to add LIO iSCSI targets. But this solution was not really portable (i.e. not running on minikube), because it requires Kernel support (LIO is an in-Kernel target).

But luckily there are some other iSCSI target implementations which provide user-space iSCSI targets - like tgtd. Alpine - which I initially used - did not offer tgtd, thus I switched over to use Debian, which provides - at least it feels like it - every single project which exists since the beginning of unix timestamp 0.

Anyhow, long story short. With tgtd it was easy to create an image which contains Alpine and CirrOS as demo images, and export them in a tgt target.

The result is now a simple docker image which you can launch, and if you export the right port, then you got a ready to use iSCSI target which serves Alpine, CirrOS, and an empty LUN. All of those are inside the contianer, thus the data is not persisted.

Feel free to try it out:

$ docker run \
  -p 3260:3260 \
  -it fabiand/iscsi-demo-target-tgtd


# To test:
# In another terminal
$ qemu-system-x86_64 \
  -snapshot \
  -serial stdio \
  -drive file=iscsi://127.0.0.1/iqn.2017-01.io.kubevirt:sn.42/2

# Or just to discover
$ iscsiadm --mode discovery -t sendtargets --portal 127.0.0.1
Feb. 28 2017
#kubernetes #debian #alpine #cirros #kubevirt #libvirtd #qemu #fedora