Automated Firmware Tests

Problem

Figure 1. A firmware gets built, remotely uploaded to a target device and remotely tested via a single command

Developing an embedded device firmware is a challenging task. It is crucial to make sure that the firmware functionality does not break as developers make changes to the code.

Continuous Integration (CI) systems such as GitHub Actions, Jenkins, Azure CI can be used to automatically rebuild firmware binaries each time code changes are submitted.

The next step is to automatically test built firmware on a real target device. But it is very rarely implemented, as is not trivial to build such a system ad-hoc. For example, there are following options:

  1. Configure a dedicated custome test workstation. One can setup a dedicated test workstation, attach a tested device via a hardware debugger like Segger or BlackMagic, and write a piece of software for remote firmware upload and test. It is possible but fragile, consumes a lot of efforts and requires significant attention.

  2. Use a commercial Embedded Board Farm. The alternative is to use one of the commercial hardware test systems: Embedded Board Farms (EBF). Such commercial solutions are quite expensive, and take a significant time to setup and maintain.

At Cesanta, we faced these problems ourselves. Our software must be built and tested on many different architectures. We developed a very easy to use framework for the Continuous Automated Firmware tests, which performs automatic hardware tests for every change in the code.

Solution: vcon.io

Our online service - vcon.io, solves this issue by providing the following:

  1. A pre-built firmware for the ESP32/ESP32C3 devices which can act as remotely managed programmer and serial monitor. By wiring such module to the target device, it is possible to reflash a target device at any time, and read/write to its serial port
  2. A secure, authenticated management cloud, https://dash.vcon.io, for managing ESP32/ESP32C3 modules. That cloud provides a simple API for reflashing, serial monitor, and more
wired module

Figure 2. Automated testing with vcon.io

To use vcon.io, you can:

  1. Use an already pre-configured test device. We provide a number of pre-wired, pre-configured, network-connected test devices that can be used immediately. You can just "rent" one of the pre-configured test devices, and spend no time on setup and management.

  2. Build your own custom Embedded Board Farm with VCON. Alternatively, it is possible to set up a test device manually, for example a test device could be your exact device. This is time consuming, but is more flexible.

In any case, with vcon.io updating a firmware remotely is as simple as uploading a firmware binary using API. In the following command, ID is the device ID, and firmware.bin - a built firmware binary for your target device:

curl -su :API_KEY https://dash.vcon.io/api/v3/devices/ID/ota --data-binary @firmware.bin

Reading serial logs is as simple:

curl -su :API_KEY https://dash.vcon.io/api/v3/devices/ID/tx?t=5

Use pre-configured test devices

wired module

Figure 3. Hardware-in-the-loop CI integration

We provide the following pre-configured devices, pre-wired and ready immediately. All devices with enabled connectivity, are connected - therefore, they can be used to test network functionality on them.

Architecture Connectivity Reflashing Serial Monitor
STM32F429ZI Ethernet yes yes
STM32F746ZG Ethernet yes yes
STM32H743ZI Ethernet yes yes
TM4C1294XL Ethernet yes yes
NXP MIMXRT1020 Ethernet yes yes
Renesas RA6M3 Ethernet yes yes
RP2040 + W5500 Ethernet yes yes
nRF9160 Cellular yes yes
RP2040 pico-w WiFi yes yes
STM32L432KC n/a yes yes
ATSAMD51 n/a yes yes
ATSAMD21 n/a yes yes
ATMEGA328P n/a yes yes
nRF52832 n/a yes yes

Note: we can add other architectures on demand, if they are not already listed. Please contact us if you'd like to go ahead with this option.

Use your own device

Using VCON, it is easy to build a custom EBF with minimal efforts and resources. Here is the action plan:

When done, your target device will have an authenticated, secure RESTful API for reflashing and capturing device output. It can be called from anywhere, for example from the software CI.

wired module

Figure 4. A RaspberryPI Pico-W5500-EVB tested using the ESP32C3 XIAO. Part of our Mongoose Networking Library automated tests

Take any ESP32 or ESP32C3 device - a devboard, a module, or your custom device. We list some of the recommended hardware in our documentation. Then,

When your device is wired, you can reflash your device remotely with a single command:

curl -su :API_KEY https://dash.vcon.io/api/v3/devices/ID/ota --data-binary @firmware.bin

Where API_KEY is the dash.vcon.io authentication key, ID is the registered device number, and firmware.bin is the name of the newly built firmware. You can get the API_KEY by clicking on the "api key" link on a dashboard. The device ID is listed in the table.

We can also capture device output with a single command:

curl -su :API_KEY https://dash.vcon.io/api/v3/devices/ID/tx?t=5

There, t=5 means wait 5 seconds while capturing UART output.

Now, we can use those two commands in any software CI platform to test a new firmware on a real device, and test device's UART output against some expected keywords.

Integrate with Github Actions

Our software CI builds a firmware image for us. What if left to do, is to add add a few extra commands that use curl utility to send a built firmware to the test board, and then capture its debug output.

A curl command requires a secret API key, which we do not want to expose to the public. The right way to go is to:

  1. Go to the project settings / Secrets / Actions
  2. Click on "New repository secret" button
  3. Give it a name, VCON_API_KEY, paste the value into a "Secret" box, click "Add secret"

One of the example projects builds firmware for the RP2040-W5500 board, so let's flash it using a curl command and a saved API key. The best way is to add a Makefile target for testing, and let Github Actions (our software CI) call it: calling make from CI

Note that we pass a VCON_API_KEY environment variable to make. Also note that we're invoking test Makefile target, which should build and test our firmware. Here is the test Makefile target: test Make target

Explanation:

This is the example output of the make test command described above:

$ make test
curl --fail ...
{"success":true,"written":59904}
curl --fail ...
3f3 2 main.c:65:main                    Ethernet: down
7d7 1 mongoose.c:6760:onstatechange     Link up
7e5 3 mongoose.c:6843:tx_dhcp_discover  DHCP discover sent
7e8 2 main.c:65:main                    Ethernet: up
81d 3 mongoose.c:6726:arp_cache_add     ARP cache: added 192.168.0.1 @ 90:5c:44:55:19:8b
822 2 mongoose.c:6752:onstatechange     READY, IP: 192.168.0.24
827 2 mongoose.c:6753:onstatechange            GW: 192.168.0.1
82d 2 mongoose.c:6755:onstatechange            Lease: 86336 sec
bc3 2 main.c:65:main                    Ethernet: up
fab 2 main.c:65:main                    Ethernet: up

Done! Now, our automatic tests ensure that the firmware can be built, that is it bootable, that it initialises the network stack correctly. This mechanism can be easily extended: just add more complex actions in your firmware binary, print the result to the UART, and check for the expected output in the test.

Contact us about automatic tests for your firmware