OpenAFS Wiki/ devel/ BuildbotWorkerHowto

How to set up a BuildBot Worker for OpenAFS

Rationale

Why would I want to? OpenAFS supports multiple platforms, but the strength of support for each of those individual platforms is directly proportional to the amount of testing done thereon. Although there are many different types of testing that can be done, one of the most basic is that of changes to the code base. A BuildBot worker allows for automated testing of each gerrit commit and is therefore a great resource to developers, most of whom do not have the time or resources to test their code on every platform prior to submission. Therefore, we encourage you to contribute a system as a build worker for a platform not already covered! You know you want to!

Approach

This is a consumer-oriented, high-level how-to to show how simple it is to set up your own build worker. This document is not intended to be platform-specific -- there are just too many different ways of doing things across different platforms.

Overview

A quick overview of the process:

  1. Acquire suitable system which will be continuously available
  2. Install the required software to build OpenAFS on the worker
  3. Verify you are able to compile OpenAFS from a git checkout on the worker
  4. Contact OpenAFS BuildBot admin for worker name and password
  5. Install the buildbot worker and verify it starts automatically

Since BuildBot workers pull all their information from the master, rather than having it pushed to them by the master, there is no need to open an incoming port on your firewall. As long as the worker can contact the master on the port you are given, you are good to go!

Hardware Requirements

  • Any system which OpenAFS currently supports
  • Enough free disk space to build OpenAFS tree (a couple gigabytes should do)

The machine does not necessarily need to be dedicated entirely to being a build worker; a build worker could run on any lightly-loaded, non-critical system subject to your security requirements, etc., of course. Some of the OpenAFS build workers are run by volunteers at their own homes, for instance.

Software Requirements

The software to build OpenAFS is platform dependent. See HowToBuildOpenAFSFromSource to install the tools to build OpenAFS.

Buildbot workers requires Python 3.5+. (Python 2.7 is still supported for legacy systems at this time.)

Your platform may already include a buildbot worker package included, otherwise use python pip to install the buildbot worker software and required dependencies.

pip install buildbot-worker

If you cannot or do not wish to install the buildbot into a site-wide location, you can also install it into the account's home directory or any other location using a tool like virtualenv.

See the (https://docs.buildbot.net/latest/manual/installation/index.html)[Buildbot installation guide] for more information on installing the Buildbot worker.

Compile OpenAFS from git

Use git to clone the openafs repo, then make sure the software will compile successfully. For Unix platforms, the typical build process is:

git clone git://git.openafs.org/openafs.git
cd openafs
sh regen.sh
./configure
make

Create buildbot account and home directory

It is recommended that you use a special-purpose non-privileged user account to run the buildbot worker. The name 'buildbot' is commonly used as the account name to run the buildbot worker.

The build master does not need to know the account's password -- in fact, the account does not necessarily need remote login privileges and may be locked or otherwise secured as long as your platform still allows you to run commands as that user.

Contact OpenAFS BuildBot Master admin for configuration details

The buildbot admin can be contacted by making a request on the openafs-devel@openafs.org mailing list.

The admin will provide the following settings:

  • A buildbot worker name
  • A buildbot worker password. Keep this secret.

Create your buildbot worker

Log in as the buildbot user. Create a sub directory called /home/buildbot/openafs. Buildbot will not touch anything outside of this directory.

As the buildbot user, create the buildbot worker configuration with the name and password provided by the buildbot admin:

buildbot-worker create-worker /home/buildbot/openafs buildbot.openafs.org <NAME> <PASSWORD>

Edit file openafs/info/admin to set your name and email address. This is the worker admin address, and will be visible from the build status page, so you may wish to munge it a bit if address-harvesting spambots are a concern.

Edit the file openafs/info/host to set a brief description of the host, such as OS, version, memory size, CPU speed, versions of relevant libraries installed.

Start build worker

As the buildbot user, manually verify the buildbot worker starts and connects to the buildbot master:

buildbot-worker start openafs

Check the logs to make sure it started properly. Check the OpenAFS BuildBot web portal to see the build progress presented graphically.

Set up startup scripts

Make sure the buildbot worker starts automatically when the host is started. This, of course is platform dependent. If you installed the buildbot worker with your package manager, the init scripts systemd files should already be installed. If you installed the buildbot worker with pip, you will need to setup init scripts or systemd files to start the worker when the host is started.

Here is an example systemd unit file to start the buildbot worker on systemd based hosts. Save as /etc/systemd/system/buildbot-worker.service

[Unit]
Description=BuildBot worker service
After=network.target

[Service]
User=buildbot
Group=buildbot
WorkingDirectory=/home/buildbot/openafs
ExecStart=/home/buildbot/venv/bin/buildbot-worker start --nodaemon

[Install]
WantedBy=multi-user.target

If the worker is running on a virtual machine, be sure the guest will be automatically restarted when the hypervisor is restarted.

Set up log rotation and various other useful settings

The buildbot.tac configuration file can be used to optimize your buildbot worker. For instance, you may want to change maxRotatedFiles so you don't end up with a bunch of logs littered about. Have a look at the BuildBot documentation if you wish, though you can expect to be able to take a fairly hands-off approach to administering your new OpenAFS build slave.

Link to current BuildBot documentation: http://buildbot.net/buildbot/docs/current/

Thanks for your help!