Automated Kernel Testing on RISC-V Hardware

At Codethink we have an increasing interest in the RISC-V architecture, and have, in past years, written several articles related to it, on subjects ranging from fixing a RISC-V kernel space Oops to running GNOME OS on SiFive hardware. We are also strong proponents of using robust testing pipelines to aid in the goal of long-term maintainability (LTM) of complex software platforms, such as the Linux kernel, advocating to align as closely to mainline as possible. For those not familiar with the concept of LTM, as it relates to software, we have written an article on Codethink's approach to it.

In the past we have written about our automated kernel testing pipeline that performed tests in QEMU (x86_64) and on actual hardware, namely a Raspberry Pi 4 (arm64). The results of these tests are provided to the Linux kernel's stable branch maintainers, and while any additional automated testing is appreciated for these common architectures, more 'exotic' configurations are just as important to test. For this reason, it was always our intention to implement a similar kernel testing pipeline for RISC-V hardware, and it was nice to see this approach confirmed as useful by Linus himself. As mentioned in previous articles, we already have several SiFive Unmatched boards, so it was the obvious choice for testing.

RISC-V Kernel Testing Pipeline

The main elements of our testing pipeline are shown in the figure below, consisting of a GitLab CI that fetches Linux kernel sources and a small patchset for RISC-V support that hasn't yet been upstreamed, that it then uses to build a .fit kernel image for the RISC-V board (although LAVA also supports the loading of the kernel, ramdisk and dtb from different binaries). The final stage of the GitLab CI involves the triggering of a LAVA job that downloads the image from the CI job artifacts, stores it into the worker's TFTP server and performs the testing of the kernel. The test automates and supervises the booting of the SiFive Unmatched board and user login. When succesfully logged into the OS and LAVA has detected the expected prompt, it then triggers the running of a script prelocated in the board's OS (Ubuntu 21.04), that initiates a set of OpenQA-controlled tests of the OS.

We use LAVA for the deployment and booting of the OS in the SiFive Unmatched board, and OpenQA to ensure that the system has booted correctly into the OS with the new kernel and the graphical interface is working as expected. The architectures of LAVA and OpenQA, and how they function, have already been described in the previously mentioned automated kernel testing pipeline article, so we won't go into them further. In that article we described how everything related to OpenQA and LAVA was located on a single laptop directly connected to the RPi4 we were testing on. However, the SiFive Unmatched is slightly louder than an RPi4, and as this kernel testing pipeline was expected to be running on a daily basis for the foreseeable future, we decided to place the RISC-V board and the components directly controlling it (the RPi4-hosted LAVA worker and power control relay) in a rack mount so it could run undisturbed in a server room. The remaining components of the pipeline were then deployed in VMs within Codethink's internal infrastructure. This deployment architecture is shown in the figure below.

On one VM, our public-facing LAVA and OpenQA servers are proxied through Traefik, while on the second VM, OpenQA workers can be created as needed. In the rack mount, as mentioned previously, we have an RPi4 acting as a controller for SiFive Unmatched board. Using the board's serial connection allows it to communicate with the LAVA worker instance, which in turn can control a USB power relay to power on/off and reset the board.

Future Plans

In general, now that we have deployed LAVA and OpenQA servers into our infrastructure, and added the ability to quickly spin up LAVA and OpenQA workers capable of interacting with different types of hardware, our aim is to extend our kernel testing capabilities for more architectures. One example of this is that we now also perform LTS kernel testing on the MIPS Creator CI20 platform. The results of this testing are available, just as they are for our RPi4 and RISC-V Unmatched testing pipelines, from our public OpenQA server. Eventually these results could be provided to the KernelCI project.