Higher quality of FOSS: How we are helping GNOME to improve their test pipeline

There is a lot to test on a modern digital product, and testing takes time. Companies often miss out on the latest fixes in features in the open source components they use, for fear of the time-consuming manual regression tests they have to do after upgrading a component such as Linux or GStreamer.

Codethink always advises our partners and customers that closely following mainline gives a competitive advantage (and we're not the only ones). To do this, you need to be backed up by comprehensive automated testing.

We can save even more effort by helping upstreams to improve their own automated testing capabilities. We recently built a new test pipeline for the GNOME project, covering parts of the desktop environment that usually prove difficult to test.

How do you test a GUI?

Volunteers at the GNOME project have worked hard and brought significant improvements to the project's build, test and release process. GNOME's CI builds a testing-only OS image with bleeding-edge versions of the many components that make up a modern desktop Linux system. As well as powering developer workstations, you'll see these components in many commercial Linux-based products.

Everyone uses Gitlab CI to test components and libraries in isolation, but how do you test that an operating system installs correctly, and simulate the mouse clicks and keypresses of a new user logging in for the first time? At the moment that falls to volunteers in the #gnome-os channel who manually test the OS images for regressions. But we can do better.

OpenQA for GNOME

The Linux world has a solution for UI testing: OpenQA, originally developed and maintained by SuSE. OpenQA allows writing tests for the whole installation process of an OS, checking console output, and also using OpenCV for fuzzy image matching of the screen. The image matching tests are called needles, and can determine if rendering occurs in the expected order and correctness. OpenQA also offers the ability to send keystrokes and mouse clicks that could be used to progress the installation process with an appropriate input or for entering passwords.

Working with GNOME, we set up openqa.gnome.org and created an initial set of tests. We're now working to integrate the tests into the gnome-build-meta repo where GNOME's many components are integrated ready for release.

Here's a screenshot from openqa.gnome.org:

Here's the gnome-build-meta merge request.

Integrating OpenQA with Gitlab CI

OpenQA is often used with dedicated worker machines, but we wanted to avoid creating a "pet" system that would require special maintenance. So after the CI pipeline has a Gitlab runner successfully build the artifacts, it then has another runner create an openQA worker instance with a unique ID. This ID ensures it can only be used by the pipeline that created it. The runner then issues a command to the openQA UI to perform the tests on its created worker, using API key and secret values configured in the Gitlab project.

The tests themselves are located in a sub-directory of the Gitlab repository, utilising needles cloned from the master branch of a separate needle Gitlab repository. When the test is executed the runner proceeds to poll the status of the openQA test job, to determine if it has 'passed' or not.

The design and layout of GNOME's user interface can evolve and change over time, and the tests will need to adapt. We would like to maintain separate branches of the tests following GNOME's 6 month release cadence, which is a workflow that the OpenQA UI doesn't yet support. The current implementation of the OpenQA UI requires the needles to be already in a particular directory, so they are cloned from the needles repository when the openQA UI container is created. The common workaround for this limitation is as follows:

• Ensure openQA UI will automatically update the needles repository on a change
• When a test fails, use the openQA UI to create the new or modified needle in the needles repo.
• Have a cron job running in the openQA UI container to pull from the master branch of the needle repository every minute, to ensure synchronicity if other users of the needle repository simultaneously modify the needles.

Another future goal is to run the OpenQA tests for every merge request. For now the tests can be manually started for specific merge requests, and can also be scheduled to run at regular intervals on the 'master' branch. Before we can enabling OpenQA for all merge requests, we'll need to speed up the build process so we don't want to add a big delay to every new MR.

We are looking forward to seeing OpenQA in action testing the latest GNOME desktop images in QEMU virtual machines, and if you are building a complex project with a graphical interface, you might want to investigate OpenQA as well.

Stay up to date on our Long Term Maintainability news

Receive our recent articles about Long Term Maintainability in your email.

Related Blog Posts:

• Written by Neill Whillans: Why aligning with open source mainline is the way to go >>
• Codethink's approach to Long-term Maintainability: Codethink contributes to CIP Super Long Term Kernel maintenance >>