This is the first blog post in a two-part series. Stay tuned for part two.
Codethink was contracted by the Sovereign Tech Fund to improve the testing infrastructure of systemd.
systemd has a large integration test suite with a custom-built test driver written in bash.
However, it's not easy for developers to get it running. It's fragile and requires ongoing maintenance to handle missing dependencies. In addition, extending it to support additional Linux distributions is a lot of work, and it being composed of large bash scripts is a barrier to entry for contribution.
The proposed solution was to rewrite it into something more maintainable, replacing bespoke bash with appropriate existing tools.
Codethink's solution is now live in production, with all of the old tests migrated and a new test framework introduced, which makes writing new tests easier and more robust.
In this two-part blog series, we will look at the project from start to finish, highlight the shortcomings of the old systemd integration testsuite, and introduce some new tooling we created along the way.
In this first post, we will introduce systemd's pre-existing integration test suite, explain how it works, and explain why the systemd community and the STF engaged Codethink to overhaul the integration testing infrastructure.
systemd's pre-existing integration test suite
Along with the unit tests kept alongside the code in src/test
,
and domain-specific tests like the network tests,
there are a range of integration tests configured in
directories matching test/TEST-??-*
where ??
is the test number and *
is the test name.
When referring to specific matching file paths,
I will be using TEST-??-*
as a placeholder for the
matching test name.
How it works
test/README.testsuite
describes how to run the integration tests.
The Makefile
runs the test.sh
script
which sources test/test-functions
for functions and variables that it extends and overrides
to customise how the test image is built and how the test is run.
The system image is built by loopback-mounting a formatted disk image and copying files from the host system.
This produces a fairly small image, and explicitly listing every file is not necessary since it's possible to list executables and libraries to install. The ELF headers of the executables and libraries allow the discovery of which other libraries are required.
This approach is used because the version of systemd you want to test has just been compiled against the ABI of your host system.
The initramfs you used to boot is reused to boot the VM.
The test.sh
scripts can add to the disk image's contents
by adding a hook function to add more files.
It's also possible to extend the initramfs to add new files with a provided initramfs stub, or create a new initramfs entirely.
The contents of test/units
are included in the test system1,
and for most of the tests this defines what commands are run.
For each test there is a corresponding service named
TEST-??-*.service
,
which usually runs a script called TEST-??-*.sh
.
Selecting which test to run is handled by
passing extra kernel command-line options
either on the nspawn command-line as additional arguments,
or when using qemu using the -append
option.
The command-line option
SYSTEMD_UNIT_PATH=/usr/lib/systemd/tests/testdata/TEST-??-*.units:/usr/lib/systemd/tests/testdata/units:
adds per-test and general purpose test units to the unit search path,
systemd.unit=TEST-??-*.service
selects which test to run,
and for TEST-64-UDEV-STORAGE
TEST_FUNCTION_NAME=$subtest
selects which subtest to run.
Test results are handled by tests writing to /
.
A test has failed if /failed
isn't empty
or no /testok
or /skipped
files exist.
/testok
is usually empty, but sometimes contains the string "OK",
the names of subtests that pass,
or an explanation of why a subtest was skipped.
/skipped
often contains an explanation why the test was skipped.
A test is only considered to be skipped if there is a /skipped
and no /testok
, so subtests can explain why they were skipped here.
There is a unit called end.service
when when the test isn't explicitly being run interactively
will shut the VM down after the test completes
and is activated by adding systemd-wants=end.service
to the kernel command-line.
It will also check the journal logs for a few significant warnings
and write them to /failed
to fail tests where the logs occur.
test/run-integration-tests.sh
can be used
to run multiple tests in sequence.
The sequence of operations for each test is as follows:
How it falls short
Makefiles and test.sh
The Makefiles are vestigial.
Every test uses the same Makefile whose only responsibility is to set
TEST_BASE_DIR
and run the test.sh
script.
The test.sh
scripts are usually small,
but test-functions
is 3455 lines long
and bash is not a programming language that provides
good abstractions for managing large amounts of code,
and it features a lot of dynamism
with a lot of side-effects so it's harder to maintain.
It's written to pass shellcheck, but that can't catch everything.
Loopback mounting
Creating system images by loopback-mounting a disk image requires root.
This is not ideal because it:
- Requires that developers wishing to test their changes trust they haven't been given a compromised source checkout, or that they haven't accidentally misconfigured something that could allow the tests to accidentally break their system by running unfamiliar code
- Makes it impossible to run in some classes of container or locked-down environments without sudo
- Can habituate you to entering passwords
- Can leave a mess of your working trees
if you have a mix of files with different owners
that you have to use
sudo rm
to clean up
Building from host binaries and using host initramfs
Copying files from the host system is a common practice for building an initramfs, but this requires extensive per-distribution customisation to keep track of all the required data and configuration files and any libraries opened with dlopen that aren't in ELF headers.
This is an ongoing amount of work that distributions opt-into in order to keep initramfs small, but they only need to support one distribution and can develop their initramfs builder together with the distribution and adapt to package changes as they happen.
Meanwhile, the systemd project needs to test multiple distributions and operates at a greater distance, so they won't get the memo when a component needs additional files to be included.
Since distributions typically build systems out of packages, it's also an approach that they have little incentive to support or collaborate on the development of.
As a result, this approach is somewhat fragile and has an increased maintenance burden that can't be shared.
It's also inconvenient for running tests on multiple distributions since it requires an entire host VM per distribution to build the guest VM.
Passing arguments with -append
Passing kernel command-line arguments with -append
and passing the kernel, initramfs and rootfs drive as options
is a feature only available when booting with QEMU.
Since systemd also develops the systemd-boot UEFI boot menu and the systemd-stub EFI stub to support booting kernels directly it isn't ideal for them to not be part of the integration test suite.
Checking results by /success
files
Since filesystems rarely come with a tool for extracting files from an offline image, it has to be loopback mounted to read the contents.
This has all of the problems of requiring root plus potentially losing any logs if the test corrupts the filesystem.
test/run-integration-tests.sh
This script only supports running integration tests sequentially but, there are enough tests that it would be beneficial to run some in parallel.
There's more work to do...
So far, we've learned that systemd has a comprehensive integration test suite. However, it had a range of problems, and STF funding provided an excellent opportunity to fix them.
Before the project began, Codethink collaborated with systemd and STF to create a plan to overhaul the test suite completely. The plan was presented in a milestone format to clarify to all parties what was intended to be delivered and when. This is covered in part 2 (coming soon), which also includes a deep dive into the architecture of the new integration test suite.
What is the Sovereign Tech Fund?
The Sovereign Tech Fund supports developing, improving, and maintaining open digital infrastructure. Its goal is to sustainably strengthen the open source ecosystem, focusing on security, resilience, technological diversity, and the people behind the code. Click here to visit their website.
-
Some units are generated. See the template unit and where it is configured for installation into the image ↩
Other Content
- A new way to develop on Linux - Part II
- GUADEC 2024
- Developing a cryptographically secure bootloader for RISC-V in Rust
- Philip Martin, Meet the Team
- A new way to develop on Linux
- RISC-V Summit Europe 2024
- Safety Frontier: A Retrospective on ELISA
- Codethink sponsors Outreachy
- The Linux kernel is a CNA - so what?
- GNOME OS + systemd-sysupdate
- Codethink has achieved ISO 9001:2015 accreditation
- Outreachy internship: Improving end-to-end testing for GNOME
- Lessons learnt from building a distributed system in Rust
- FOSDEM 2024
- Introducing Web UI QAnvas and new features of Quality Assurance Daemon
- Outreachy: Supporting the open source community through mentorship programmes
- Using Git LFS and fast-import together
- Testing in a Box: Streamlining Embedded Systems Testing
- SDV Europe: What Codethink has planned
- How do Hardware Security Modules impact the automotive sector? The final blog in a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part two of a three part discussion
- How do Hardware Security Modules impact the automotive sector? Part one of a three part discussion
- Automated Kernel Testing on RISC-V Hardware
- Automated end-to-end testing for Android Automotive on Hardware
- GUADEC 2023
- Embedded Open Source Summit 2023
- RISC-V: Exploring a Bug in Stack Unwinding
- Adding RISC-V Vector Cryptography Extension support to QEMU
- Introducing Our New Open-Source Tool: Quality Assurance Daemon
- Long Term Maintainability
- FOSDEM 2023
- Think before you Pip
- BuildStream 2.0 is here, just in time for the holidays!
- A Valuable & Comprehensive Firmware Code Review by Codethink
- GNOME OS & Atomic Upgrades on the PinePhone
- Flathub-Codethink Collaboration
- Codethink proudly sponsors GUADEC 2022
- Tracking Down an Obscure Reproducibility Bug in glibc
- Web app test automation with `cdt`
- FOSDEM Testing and Automation talk
- Protecting your project from dependency access problems
- Porting GNOME OS to Microchip's PolarFire Icicle Kit
- YAML Schemas: Validating Data without Writing Code
- Deterministic Construction Service
- Codethink becomes a Microchip Design Partner
- Hamsa: Using an NVIDIA Jetson Development Kit to create a fully open-source Robot Nano Hand
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Full archive