How Continuous Testing Helps OEMs Navigate UNECE R155/156

At Codethink we work with automotive OEMs on software integration strategies, helping them satisfy regulatory and market requirements. As you may expect we are seeing more organisations taking UNECE 155/156 compliance very seriously.

The upshot of UNECE R155/156 is that OEMs need to take a long-term view of the vehicles they're putting into the field. They must be prepared to provide full system software updates from the start of development to 15+ years after the last model rolls off the production line.

To succeed OEMs need the following:

• Automated and effective integration testing
• A rock-solid OTA update mechanism
• Always up-to-date software, thanks to a long-term maintainability strategy

If OEMs fail to deliver on the above they will fail to meet UNECE R155/156, with terrible business consequences. For OEMs that successfully meet the regulatory requirements the benefits include access to new marketplaces across the globe, improved automotive cybersecurity, and increased consumer confidence and trust. We want our clients to achieve this level of success, but it doesn’t happen overnight.

This article will explain why each topic is essential to ensure compliance with UNECE R155/156.

Automated and effective testing at the integration level

What do we mean by ‘at the integration level’?

Those who care about UNECE R155/156 should only be interested in testing at the full system level. I've lost count of my discussions with clients who believe unit tests alone are a valid way to measure confidence when deploying software. As an integrator, I don't care about unit tests; without a way to test the interactions of the 'units' in my integrated system, they’re worthless.

If you don't believe me, ask NASA and Lockheed Martin, who employed different measurement systems (imperial and metric, respectively) when collaborating on a Mars-bound spacecraft project in the 1990s. An assumption was made that an effective conversion between these systems was implemented, but it wasn’t covered by integration tests, resulting in a factor 4.45 error in thrust, leading to disaster. The unit tests all passed.

Automating your integration tests is more complex and costly than unit tests. Ultimately this work needs to be done, and it needs to be established as early as possible in the life of a project. Although manual testing can never be entirely replaced (and shouldn't be), Codethink has worked with multiple OEMs to establish automated integration tests using entirely open source tooling (Testing in a Box, USB-Switcher, OpenQA) both at the level of individual ECUs and the combined ECU full vehicle release level.

Implementing a rock-solid update mechanism

It goes without saying, but it’s critically important to have an over-the-air software update that is 100% reliable. It needs to be bullet-proof, and it needs to be tested to the point at which you have complete confidence in it being bullet-proof. Every project understands the former. Sadly, many projects do not understand the latter. If they did, the OTA tests would be one of the very first automated tests written (or at least written as soon as feasible), and the devices under test would be flashed/updated via OTA in every single CI run.

One of the worst possible scenarios for an OEM is performing an OTA rollout, crossing your fingers, and hoping it goes well. This is why Codethink advocates automating the test as soon as it’s feasible. We have recently worked with an OEM - and collaborated with various third-party suppliers - to implement this, achieving a fully automated end-to-end test of an OTA update via GitLab CI pipelines. This consisted of taking a release candidate image, triggering the fleet management platform API to perform a rollout action, connecting to the telematics ECU and pushing the latest update to the IVI system.

Compared with the previously all-manual OTA testing, this had a 10X impact on the capacity to test software rollouts, allowing thousands of OTA tests to be completed before any production update.

Why up-to-date software matters

Many readers of this blog post will already be aware that Codethink considers the current status quo for deploying software indefensible. We have spoken about this before, of course. Traditionally, OEMs were happy to release vehicles into the field without an upgrade plan, patching them only when necessary, effectively keeping them on 'life support'. It should by now be well established that the scale of software complexity we're dealing with extends far beyond this.

OEMs who want to comply with UNECE R155/156 must plan to deploy fully upgradeable software. For the vast amount of open source software in use this means keeping as close to upstream releases as possible. This is nothing new, of course, and it’s not just Codethink that has been saying it.

For commercial software OEMs need to have contractual terms with suppliers to ensure they will continue to support over the lifetime of the vehicle production. This could be either via updates to old versions or commitments to provide the next version of the software in an easily consumable manner (i.e. small incremental updates rather than significant breaking changes years apart). In an ideal world OEMs could even persuade suppliers to provide source code and/or push their changes upstream instead of agreeing to give updates over the long term. OEMs who care about compliance with UNECE R155/156 should be pushing for source code as it grants the level of control needed for the OEM.

Regardless of whether the software is proprietary or FOSS, the biggest challenge OEMs face when updating software is the vast amount of regression testing needed to instil confidence before a complete system upgrade. This is where automated testing for vehicle security steps in: the more automated tests in place earlier in the project, the more valuable testing can be done every time an update occurs, instilling more confidence and accelerating the development process.

Conclusion

Ultimately for OEMs to navigate the UNECE R155/156 compliance minefield, continuous testing in automotive is not just a best practice but a necessity. The regulations demand a long-term commitment to software integration, updates, and long-term maintainability.

OEMs can help to ensure compliance throughout a vehicle's lifecycle by focusing on automated integration testing and robust OTA update mechanisms. In addition, demanding accountability for long-term maintainability from suppliers is vital to meeting regulatory challenges.

With the right processes and strategies OEMs will ensure their vehicles remain reliable and compliant for years to come.

At Codethink, we help our customers overcome safety and security challenges to deliver critical software projects. Click here to learn more about our approach to delivering trustable software.