At Atomic, we build a lot of systems that incorporate device firmware talking to software. We also commonly interface with other firmware or software teams.
I believe every project that involves firmware talking to software should include a high-level suite of tests that are written against the firmware’s interface to the software (e.g. Bluetooth, Ethernet, USB, etc). The tests should exercise the complete set of functionality that is exposed to the software. Writing these tests is not always straight forward because they commonly require human interaction with the physical device, but they are invaluable to a product team.
Unfortunately, these types of test suites are not always prioritized by product managers. They should be, and here is why.
1) It proves that the firmware works.
The main purpose of the test suite is to prove that the firmware and hardware function properly. Ideally, the test suite should be decoupled from the software product and created by the firmware team. These tests lead to a lot less finger pointing between the software and firmware teams.
2) It’s easier to debug the complete system.
When issues arise in a complex system, being able to isolate and reproduce the problem is the first step to resolving them. Having a set of high-level firmware tests allows the team to quick determine if the issue is caused by the firmware or software. For the same reason, it’s also extremely valuable for the software to have a hardware/firmware simulator, but I’ll save that discussion for another post.
3) It allows less-technical team members to test the firmware.
Very few people are skilled enough to read a protocol analyzer or shove a series of bytes into a firmware program. But lots of people are smart enough, with a little training, to set up and run a firmware test suite. The ability to get firmware testing time out of non-firmware engineers is huge leverage for any team.
In order to make your firmware test suite more realistic and not a complete nightmare to author, you should leverage the fact that you’ll have a smart person running the test suite. When you need to interact with a device (e.g. push a button, plug in a cord, etc), simply prompt the human to do that step and then continue the automated suite. In some cases, you may even require them to physically observe a change on the hardware and report their observations before continuing.
4) It enables regression testing.
During the product lifecycle, you’ll most likely rev your hardware or firmware. When that happens, the firmware test suite will be your best friend for the validation process.
5) It gives you a leg up on production testing.
Testing firmware during the manufacturing process is an engineering best practice. I’ve witnessed portions of firmware test suites being reused for end-of-production-line hardware device testing. Having the firmware test suite in place before you begin the manufacturing process gives you a head start on that effort.
6) It helps with field testing.
If you have a sophisticated hardware product, chances are you’ll need to provide some onsite or remote support. Having a firmware test suite will allow you to quickly identify issues with the hardware in the field.
If you have a war story about how a firmware test suite saved your life during a project, or a story about how the lack of such a test suite caused you pain, please share it in the comments below.