A new way to develop on Linux - Part II

In our last update, we shared how we can improve the developer experience for building and testing software by using system extensions with sysext-utils. This time, we want to share how we can leverage that work to improve end-to-end testing of operating systems.

Testing with GNOME OS

For a couple of years now, folks at GNOME have been using openQA with GNOME OS to run end-to-end tests for different components of their stack; from the installer, to the shell, to many of their core applications. These tests run every time changes land in GNOME OS.

Although these tests have helped identify bugs early on, the current workflow comes with some disadvantages and costs.

Every time a change lands in GNOME OS and tests are run, an entire operating system has to be built from scratch. This is slow and uses valuable resources. Since it all happens after the changes have already been merged into a GNOME component, any problems detected by these tests will only be discovered after the fact.

Plus, people interested in these changes must wait until a new GNOME OS build is available.

Leveraging system extensions

System extensions allow us to build and overlay individual software components on existing system images. We can then test the new component without rebuilding the entire OS.

As part of our collaboration with the GNOME Foundation, through the Sovereign Tech Found (STF), we explored the use of extensions to tackle these disadvantages and costs with the current workflow. Using GNOME Shell as a test case, we came up with a new workflow that goes like this:

1. A developer sends a merge request to gnome-shell.
2. A CI job uses sysext-utils to build a system extension out of that gnome-shell merge request and publishes the extension.
3. Another CI job sets up openQA to run the end-to-end tests on the latest GNOME OS build, with that system extension overlaid.

The benefits of new this approach are:

• There is no need to build an entire operating system to test the changes, only the component that changed.
• Tests are run before the changes are merged.
• The published extension can be used by reviewers to manually try these changes in GNOME OS using sysext-utils. This could also be useful for design review and manual QA testing.

This workflow is inspired by Flatpak workflows used in places like Flathub, which is beneficial in itself, considering that many GNOME developers are already familiar with that workflow.

To achieve this we developed new GitLab CI/CD components for building and testing extensions, easily available to all GNOME modules. In fact, we have already integrated some of these jobs in mutter and gnome-shell.

Future work

There are a few merge requests pending review, so things might still change a bit. Nevertheless, we have demonstrated that it's feasible to integrate system extensions with development and testing pipelines, and that we can provide an improved workflow for reviewers and testers, through that integration.

During the process of designing and building this workflow, we identified many opportunities for future improvement. For example:

• Extensions can be better integrated with openQA. Its current API has no concept of system extensions and, therefore, custom SMBIOS wizardry is still needed to integrate extensions.
• Reduce test duplication. Moving openQA tests to each individual GNOME module results in common tests being duplicated across multiple projects and, therefore, reorganisation is needed to reduce duplication.

Additionally, this work could be extended and reused in non-GNOME system components as well, like Flatpak and XDG Desktop Portal, and even other Linux distributions like Fedora by providing similar CI/CD components.

If you’d like to learn more about our work with GNOME, systemd, or anything discussed in this article, please contact us via sales@codethink.co.uk.