Improving systemd’s integration testing infrastructure (part 1)

This is the first blog post in a two-part series. Click here to read Part 2.

Codethink was contracted by the Sovereign Tech Fund to improve the testing infrastructure of systemd.

systemd has a large integration test suite with a custom-built test driver written in bash.

However, it's not easy for developers to get it running. It's fragile and requires ongoing maintenance to handle missing dependencies. In addition, extending it to support additional Linux distributions is a lot of work, and it being composed of large bash scripts is a barrier to entry for contribution.

The proposed solution was to rewrite it into something more maintainable, replacing bespoke bash with appropriate existing tools.

Codethink's solution is now live in production, with all of the old tests migrated and a new test framework introduced, which makes writing new tests easier and more robust.

In this two-part blog series, we will look at the project from start to finish, highlight the shortcomings of the old systemd integration testsuite, and introduce some new tooling we created along the way.

In this first post, we will introduce systemd's pre-existing integration test suite, explain how it works, and explain why the systemd community and the STF engaged Codethink to overhaul the integration testing infrastructure.

systemd's pre-existing integration test suite

Along with the unit tests kept alongside the code in src/test, and domain-specific tests like the network tests, there are a range of integration tests configured in directories matching test/TEST-??-* where ?? is the test number and * is the test name.

When referring to specific matching file paths, I will be using TEST-??-* as a placeholder for the matching test name.

How it works

test/README.testsuite describes how to run the integration tests. The Makefile runs the test.sh script which sources test/test-functions for functions and variables that it extends and overrides to customise how the test image is built and how the test is run.

The system image is built by loopback-mounting a formatted disk image and copying files from the host system.

This produces a fairly small image, and explicitly listing every file is not necessary since it's possible to list executables and libraries to install. The ELF headers of the executables and libraries allow the discovery of which other libraries are required.

This approach is used because the version of systemd you want to test has just been compiled against the ABI of your host system.

The initramfs you used to boot is reused to boot the VM.

The test.sh scripts can add to the disk image's contents by adding a hook function to add more files.

It's also possible to extend the initramfs to add new files with a provided initramfs stub, or create a new initramfs entirely.

The contents of test/units are included in the test system¹, and for most of the tests this defines what commands are run. For each test there is a corresponding service named TEST-??-*.service, which usually runs a script called TEST-??-*.sh.

Selecting which test to run is handled by passing extra kernel command-line options either on the nspawn command-line as additional arguments, or when using qemu using the -append option.

The command-line option SYSTEMD_UNIT_PATH=/usr/lib/systemd/tests/testdata/TEST-??-*.units:/usr/lib/systemd/tests/testdata/units: adds per-test and general purpose test units to the unit search path, systemd.unit=TEST-??-*.service selects which test to run, and for TEST-64-UDEV-STORAGE TEST_FUNCTION_NAME=$subtest selects which subtest to run.

Test results are handled by tests writing to /. A test has failed if /failed isn't empty or no /testok or /skipped files exist.

/testok is usually empty, but sometimes contains the string "OK", the names of subtests that pass, or an explanation of why a subtest was skipped.

/skipped often contains an explanation why the test was skipped. A test is only considered to be skipped if there is a /skipped and no /testok, so subtests can explain why they were skipped here.

There is a unit called end.service when when the test isn't explicitly being run interactively will shut the VM down after the test completes and is activated by adding systemd-wants=end.service to the kernel command-line.

It will also check the journal logs for a few significant warnings and write them to /failed to fail tests where the logs occur.

test/run-integration-tests.sh can be used to run multiple tests in sequence.

The sequence of operations for each test is as follows:

How it falls short

Makefiles and test.sh

The Makefiles are vestigial. Every test uses the same Makefile whose only responsibility is to set TEST_BASE_DIR and run the test.sh script.

The test.sh scripts are usually small, but test-functions is 3455 lines long and bash is not a programming language that provides good abstractions for managing large amounts of code, and it features a lot of dynamism with a lot of side-effects so it's harder to maintain.

It's written to pass shellcheck, but that can't catch everything.

Loopback mounting

Creating system images by loopback-mounting a disk image requires root.

This is not ideal because it:

1. Requires that developers wishing to test their changes trust they haven't been given a compromised source checkout, or that they haven't accidentally misconfigured something that could allow the tests to accidentally break their system by running unfamiliar code
2. Makes it impossible to run in some classes of container or locked-down environments without sudo
3. Can habituate you to entering passwords
4. Can leave a mess of your working trees if you have a mix of files with different owners that you have to use sudo rm to clean up

Building from host binaries and using host initramfs

Copying files from the host system is a common practice for building an initramfs, but this requires extensive per-distribution customisation to keep track of all the required data and configuration files and any libraries opened with dlopen that aren't in ELF headers.

This is an ongoing amount of work that distributions opt-into in order to keep initramfs small, but they only need to support one distribution and can develop their initramfs builder together with the distribution and adapt to package changes as they happen.

Meanwhile, the systemd project needs to test multiple distributions and operates at a greater distance, so they won't get the memo when a component needs additional files to be included.

Since distributions typically build systems out of packages, it's also an approach that they have little incentive to support or collaborate on the development of.

As a result, this approach is somewhat fragile and has an increased maintenance burden that can't be shared.

It's also inconvenient for running tests on multiple distributions since it requires an entire host VM per distribution to build the guest VM.

Passing arguments with -append

Passing kernel command-line arguments with -append and passing the kernel, initramfs and rootfs drive as options is a feature only available when booting with QEMU.

Since systemd also develops the systemd-boot UEFI boot menu and the systemd-stub EFI stub to support booting kernels directly it isn't ideal for them to not be part of the integration test suite.

Checking results by /success files

Since filesystems rarely come with a tool for extracting files from an offline image, it has to be loopback mounted to read the contents.

This has all of the problems of requiring root plus potentially losing any logs if the test corrupts the filesystem.

test/run-integration-tests.sh

This script only supports running integration tests sequentially but, there are enough tests that it would be beneficial to run some in parallel.

There's more work to do...

So far, we've learned that systemd has a comprehensive integration test suite. However, it had a range of problems, and STF funding provided an excellent opportunity to fix them.

Before the project began, Codethink collaborated with systemd and STF to create a plan to overhaul the test suite completely. The plan was presented in a milestone format to clarify to all parties what was intended to be delivered and when. This is covered in Part 2, which also includes a deep dive into the architecture of the new integration test suite.

What is the Sovereign Tech Fund?

The Sovereign Tech Fund supports developing, improving, and maintaining open digital infrastructure. Its goal is to sustainably strengthen the open source ecosystem, focusing on security, resilience, technological diversity, and the people behind the code. Click here to visit their website.