oVirt Node and Foreman reminder

netbulae noticed, that the following snippet can be used during an oVirt Node auto-installation to prevent continous reboots:

local_boot_trigger=http://foreman.example.com/unattended/built?token=<%= @host.token %>

That will tell Node to call this URL after the auto-install, calling that URL tells Foreman to disable PXE boto for that host.

Reminder: Take care to adjust the FQDN of your foeman instance in that URL above.

SuSE logo

Seeing our always popular-to-be-bought friend SuSE beeing sold again. I wonder what’s left of the it’s core and values …

A beautiful renderer, seen in action at https://graphhopper.com/maps/ - which is quite good btw.

Markdown and footnotes

It seems that footnotes can be easily used in markdown (yey!):

_Needless to say[^say] to much._
[^say]: Everything has been said.


Needless to say1 to much.

  1. Everything has been said. 



Persistence is a somewhat relevant topic for Node.1

When I say persistence I’m talking about keeping the state of files between images. And the two aspects I want to speak about is how files are persisted and what files are persisted.

Node is delivered as an image (now and in future). An update (a new rootfs) is also delivered as an image. Migrating changes between the original image, and the updated image is what I call persistence.


In the current Node persistence is implemented by keeping changed files on a writable partition, and bind mounting them at boot time into the target. The persistence works with a white-list, so a user (or package) needs to name all files which need to be persisted. A drawback of this approach is, that the bind-mounts are not available in the early boto stages. The problem can be illustrated if you think about persisting a modified /etc/fstab or new (3rd party) kernel modules. Additionally someone needs to determine the files which define the state of a component.


For the future we are thinking about an approach where the whole root filesystem is writable2.

In addition to it beeing writable, we will be also booting into that filesystem. This makes it possible to

  • Modify any file
  • (which implies) Modify files which are relevant for the early boot process

The key difference to the old approach is, that we can now modify the filesystem we use to boot.


Persistence then works by migrating a file from the previous to the next image. The migration can be a simple copy or similar to a 3-way merge. Maybe even both. Or more? Independent of what strategy is choosen, he key fact remains: Persisting works by migrating the old state, to the new image.

Now we know how we persist files, but it’s still the question what files we need to be persist.


As said above, currently this works by whitelisting files. This has the drawback that the user (or package/consumer) needs to be aware that it#s running on Node, and that is cumbersome to maintain, and not obvious to a user, why files need to be persisted.

I anticipate something like “transparent” persistence for the next-gen Node. It should be transparent to the user/consumer that persistence is necessary, so Node should decide what files need to be persisted which not.

Because Node will keep the original image, and a layer with the changed filesystem tree around, we can easily determine the changed files.


A clear option is to migrate all of the changed files to a new (updated) image. This approach is greedy: Each file that has been touched is copied from one image to the next. This will result in many changes which are moved from one image to another. We need something smarter. And there is more than one option.


We could use file-level white-listing like we do today.

$ perists /etc/passwd

It’s very selective, but needs a lot of maintenance.


Besides doing it on a file level, we can also do it on the package level. Two approaches: We can either name the package (and all of it’s files) to be persisted, or we can determine the package of a file, when the user tries to persist a file.

$ persist pkg vdsm

$ persist file /etc/passwd
Do you want to perists all changes to files of the package 'core-utils' instead of the single file '/etc/passwd'?
Yes/No/Maybe [Maybe]?


Or we could just come up with a blacklist of paths which should not be persisted. Except when explicitly requested.

These are a couple of options that come to my mind, I’m not yet sure how the final solution will look like. But I see use cases for all this ways. And the good is, the current infrastructure we plan, will support all of these approaches.

  1. it’s not only Node that needs persistence. It affects all projects which need to take care to keep the customized state of a system. If the state is not kept, then we are talking about stateless

  2. Currently the root filesystem is read-only, and only parts are made writable using tmpfs and bind-mounts to writable partitions 

For long it was on my list to gget started with libreswan (part of Fedora) and created some ipsec tunnel between hosts.

It turns out that configuring libreswan is much easier than configuring the kame ipsec tools.

The link above is a good starting point, it can also be applied to arch.

Now I just need to find a way of how to distribute a key via mdns.

Node is moving.


We visited project atomic and the btrfs idea of the systemd cabal. The first does not really fit and has issues in our usecase, the latter is very bleeding edge. We visited them to evaluate them for the next-gen Node.

But before we come to approaches, let’s take a brief look at some issues. Node is everything. An image, an installer, a configuration UI, a way of how a system can run. And that makes it hard to develop, maintain, test and cumbersome to use. But I should note that it is doing it’s job well, beeing stable at runtime.

We believe that Node can be improved if we delegate responsibilities to individual (existing) components. A drawback of this will be that removing the high integration and going over to a more modular approach will likely increase the size of the currently very small (~250MB) image.

In the next generation of Node, we are looking at the components to be used in parts of Node

  • anaconda to replace our own installer,
  • puppet to do the system configuration (locally and central) instead of using our own classes,
  • livemedia-creator (oz/imagefactory) to build the appliance (and iso), and
  • imgbased to take care of persistence and multiple-images.

It is not yet set in stone. But the investigations we do are looking promising. And not everything will change: Likely we’ll keep our ovirt-node-setup TUI (but we are also having an eye on anaconda’s initial-setup).

The outcome we expect is

  • less installer problems and maintenance burden as we rely on anaconda
  • less configuraiton bugs as we reuse the well maintained puppet classes, and can also contribute to them
  • less build problems because livemdia-creator is running the builds in isolated VMs
  • better runtime experience for users, because the filesystem will be writable
  • easier to integrate 3rd parties, because we aim to make the persistence transparent
  • better testability by delegating responsibilities
  • more distro agnostic

The biggest drawback will be the increased size. We don’t know yet where we will land, but lets throw 1 GB into the air. We will also - obviously - be much more dependent on other components - but we don’t fear that.

Much to come, and we are just starting. If you want to join, reach out to us on irc://irc.oftc.net/#ovirt or irc://irc.freenode.net/#ovirt-node.

“Weint eigentlich noch jemand der Glühbirne hinterher? Ich jedenfalls nicht, angesichts der rasanten Fortschritte der LED-Hersteller. Als ich vor vier oder fünf Jahren meine erste LED-Birne für stolze 35 Euro gekauft habe, kam ich mir noch ziemlich verarscht vor: Das Ding verbrauchte zwar kaum Strom, produzierte dafür aber auch kaum Licht. Nicht einmal als Leselampe taugte es etwas. Heute habe ich wunderbare 6-Watt-LEDs für unter zehn Euro, die ähnlich warmes Licht abgeben wie eine herkömmliche matte 40-Watt-Birne. Ob es ohne Glühbirnenverbot eine ähnliche Entwicklung gegeben hätte? Darüber kann ich nur spekulieren. Geschadet haben dürfte es jedenfalls nicht.”

Gerrit’s REST API

Gerrit has a nice REST API.

I had the need to quickly check if a change was merged or not, it turned out to be quite easy:

while read CHANGE ; do curl "gerrit.ovirt.org/changes/ovirt-node~master~$CHANGE" ; done

That way I just had to paste the change-id into the terminal and quickly got my response.

bookzilla instead of amazon

This is nothing new, actually uncovered last year already. But today I just saw it again, and some aspects are just not what I want to support with my kaufkraft/buying power. And that is actually our - consumers - best weapon: Where we spend our money.

So, time to re-activate my bookzilla account, which is also supporting F/LOSS.


There is surely the possibility of using those sub-shops of amazon which also support F/LOSS, but those sub-shops will still use amazon’s infrastructure and “huma ressources”, which ain’t what I want.

But I must admit that I am not sure how the workers at libri/bookzilla are treated.

This .travis.yml actually shows how easy CI can be — on simple packages, nothing like (Node)[http://www.ovirt.org/Node].

  - curl http://www.rust-lang.org/rustup.sh | sudo sh -
  - cargo build --verbose
  - cargo test --verbose
  - rustdoc --test src/lib.rs -L target
  - LD_LIBRARY_PATH=/usr/local/lib

Source: https://github.com/alexcrichton/rust-compress/blob/master/.travis.yml

Missing Computing Device Manufaktur

Goldene Herrentaschenuhr Nr. 1613

I’m not really sad, but I wonder: Where are the people and companies caring about decent computing hardware. I mean shiny, stylish laptops, solidly build, reliable, durable, and maybe not beef, but tofu.

We know that one … who was it … vendor. But that can not be everything. Many others make devices, but they lack it - the bit which makes a device perfect, different, fitting, the thing you want.

Can it be that hard?

The mobile world is showing that smalld evices can also be done by small “Manufakturen”. Geekpshone, Jolla, and Fairphone are just three examples of small vendors which build decent hardware.

The Anaconda installer is a piece of software used to install Fedora, RHEL and their derivatives. Since the installation is internally a complicated process and there are many aspects of the resulting system that need to be configured during installation it has been decided that the Anaconda installer needs to support dynamically loaded plugins (addons) that will be developed and maintained by other teams and people specialized in certain areas. This guide should help those people writting an Anaconda addon to get an insight to the architecture and basic principles used by the Anaconda installer as well as an overview of the API and helper functions provided.

Ever wondered how to write addon’s for anaconda? Vratislav wrote it down for all of us!