Buildbot Linux RC builder for OpenAFS

Most releases of Linux require OpenAFS code changes, since OpenAFS is out of tree due to licensing incompatibilities between GPL and the IBM public license. In order to catch build errors early, a nightly builder on the OpenAFS buildbot has been created to automatically build the various git branches of OpenAFS on the most recent Linux kernel release candidate every night. The builder spins up a virtual machine guest, installs the most recent Linux release candidate image and headers, reboots the guest to run the new Linux release candidate, then attempts to build and run OpenAFS on the release candidate.

Linux KVM and libvirt commands are used for virtualization. The builder downloads pre-build Linux release candidates published by the Ubuntu Kernel Team.

The components of this system are:

A Linux KVM host system (kvm, qemu, and libvirt)
An Ubuntu base image to be cloned
A script called kernel-ppa to find and download the most recent Linux kernels from the Ubuntu Kernel Team's site.
A script called virt-run to clone the base image, run commands on the clone, then destroy the clone
A buildbot worker which is always running
The buildbot master configuration

Linux KVM host system

The host system is currently configured to use Linux KVM to create ephemeral virtual machines to run the build and tests. The host system must have cpu virtualization extensions (Intel VT-x or AMD-V) to run KVM.

See your distribution specific documentation for KVM and libvirt installation instructions.

Ubuntu base image

Create a base image to be cloned by the buildbot worker. The base image can be created with the libvirt-manager GUI program. The OpenAFS buildbot is currently using an Ubuntu 18.04 LTS release image.

Install the afsutil package from python pip. This command is used to install the built binaries and create a test cell on ephemeral guests.

sudo pip install afsutil

Install the build tool chain needed to build OpenAFS from source using sudo afsutil getdeps or by running apt-get install of the packages needed. At this point it is best to verify the OpenAFS binaries can be built from a git checkout on a running base image.

The kernel-ppa script is used to find and install the most recent Linux kernel image and headers from the Ubuntu Kernel Team's site. Download kernel-ppa from github and copy to a location in the path on the base image.

git clone https://github.com/meffie/virt-utils.git
cd virt-utils
git checkout virt-run
sudo cp kernel-ppa /usr/local/bin/

Allow the buildbot user on the guest to run kernel-ppa and afsutil as sudo, without a password, in order to change the kernel on ephemeral guests.

$ cat /etc/sudoers.d/buildbot
buildbot ALL=(ALL) NOPASSWD: /usr/local/bin/afsutil
buildbot ALL=(ALL) NOPASSWD: /usr/local/bin/kernel-ppa

Keep the base image guest running until the buildbot worker's ssh public key is installed (see the next section). After which, the base image guest should be shutdown, otherwise libvirt will not be able to clone the base image when a build is scheduled.

Buildbot worker

The buildbot worker should be running on the host system or in a long-running virtual machine.

Create a 'buildbot' service account on the worker system and create an ssh key-pair for the communication from the buildbot worker to the ephemeral virtual machines. The ssh-key should be password-less for the automation. Copy the public ssh key to the buildbot user's authorized keys file on the running base image guest. After the key is in place and verified to work, shutdown the base image guest so it can be cloned.

Configure the account to have access to libvirt commands by setting the environment variable LIBVIRT_DEFAULT_URI. See the KVM and libvirt documentation.

Install the virt-run command from the 'virt-utils' project on github.

git clone https://github.com/meffie/virt-utils.git
cd virt-utils
git checkout virt-run
sudo make install

Verify the virt-run command is able to clone the base-image to spin up an ephemeral guest and run commands on it. See the examples directory in virt-utils for some example usage.

Finally, install the buildbot worker from your distibution packages or from python pip and configure the slave to contact the buildbot master.

Buildbot master configuration

Changes to the buildbot master can be deployed by one of the OpenAFS buildbot admins. The buildbot master configuration is maintained in the OpenAFS-Contrib project on github.

Currently, the buildbot master is configured to virt-run to spin up a clone, install the current linux release candidate, reboot the clone, and then build OpenAFS.