Securing the boot sequence in embedded Linux devices

With IoT, 5G, and embedded devices becoming a big part of everyone’s daily lives, security should be on everyone’s minds. Most importantly, security and trust in your embedded devices are essential for many reasons that we’ll look at in the post. However, embedded devices have not always had good security with the last several years seeing a significant number of high-profile hacks that should give most pause before widely adopting IoT in their homes and other elements of their lives.

Security in Linux devices is quite a large topic. In this post, we’ll specifically drill down into how secure boot works on embedded Linux devices. First, we’ll talk about why boot security is essential. Then we’ll describe what it is in the context of embedded Linux, and finally, we’ll show you how Pantavisor implements secure boot in a containerized embedded Linux system.

Why is boot security important?

Boot security is a fundamental security layer that ensures our devices are running the software we intend them to run. Without boot security, a compromised device can have its kernel or applications modified to run malicious code. Without boot security, even after restarting the device, it could still be compromised. In the worst case, unauthorized code changes can potentially refuse remote updates to compromise the device permanently. This lack of security may be inconvenient for small devices such as smart light bulbs or it can be financially expensive if items such as home appliances or automobiles are compromised. These are not abstract and theoretical concerns, as attacks on the boot process have been and continue to be seen in the wild.

What is Secure Boot in Embedded Linux?

Secure boot, in essence, is a process that ensures a device boots and operates using software that the OEM trusts. This is important and, surprisingly, is not a standard configuration for your laptop or on many IoT devices. The secure boot process uses cryptographic techniques, namely public keys, signatures, and hashes, to ensure that the system’s software components are specified initially. At a basic level, secure boot depends on the concept referred to as a ‘chain of trust.’

Chain of trust in a boot sequence

The boot process for embedded Linux consists of a distinct series of steps. The fundamental concept of the chain of trust is that each step in the process verifies that the subsequent step is valid and unmodified before initiating that step. In the case of a failed verification on any of the steps, the boot process fails, and the device will not complete its start-up.

The main steps in booting an embedded Linux device are: executing the ROM code, running the boot loader, starting the kernel, and finally mounting the root file system. As mentioned, each step is responsible for validating the next step before continuing.


Chain of Trust

ROM Code

The ROM code is key to kicking off the whole process. The board manufacturer provides the ROM code, and its job is to verify and launch the boot loader. It is cryptographically signed in boards that support secure boot and is considered the root of trust as it is the first step. While this stage is generally secure, it is not unheard of for the ROM code itself to have security vulnerabilities, so the prudent developer should stay aware of the current critical issue status of boards they are developing on.

In the secure boot process, the boot loader executable will be signed. The ROM code will have a public key to confirm the signature of the boot loader before launching it. The boot loader public key’s storage location depends on the board implementation but could be in for example, One Time Programmable memory(OTP) or Trusted Platform Module hardware (TPM). Once the ROM confirms the bootloader signature, it will be launched.

Boot Loader

The boot loader is the next step in the chain of trust of the secure boot process. Its job is to verify the Linux kernel executable and launch it. U-Boot is an example of a boot loader used on many embedded Linux systems. As in the previous link in the chain of trust, the kernel is signed. The boot loader file contains the public key to verify the kernel file before launching it. Since the ROM code has already verified the boot loader file, we can be sure that its public key for the kernel is correct. Once the kernel signature is verified, the kernel is launched, otherwise the boot process aborts. One key point is that each stage of the secure boot process can only use already verified resources, which at this point limits the bootloader to itself and the device ROM.


The kernel of course is the core of the Linux OS. Its responsibility in the chain of trust is to verify the root file system. As the root file system is usually significantly larger than the boot loader file or the kernel file, it is not practical to cryptographically sign the file system as a whole. Instead, it is typically broken down into blocks, each of which are hashed into a tree that overall contains the whole file system validity in parts. The parts can then be verified independently as required when the blocks are loaded. The kernel module dm-verity (Device Mapper Verity) is often used for this role.

Root File System

Once the root file system is mounted, the initrd process can be launched as the first process in the system.

The secure boot process is now complete and the system is up and running. We can now be sure that all of the components loaded during the boot process have been verified by the previous step in the secure boot chain of trust sequence.

Pantavisor and Secure Boot

In the previous section, we described in broad strokes how secure boot works in embedded Linux systems. In this section, we’ll focus on how Pantavisor implements and supports secure boot on embedded Linux systems. Pantavisor supports the ability to protect runtime code and configs of both apps and BSPs using cryptographic signatures that get enforced at both install and runtimes.

A key difference with Pantavisor is that it can support multiple versions of the device definition that we refer to as Steps. Each Step is a single device revision including all of the objects, containers and system assets that are part of a specific firmware revision. The revision history of Steps is referred to as a Trail. Having a series of revisions available means that you can easily roll back the device to an earlier ‘good state’ should an update or a new rollout fails. The Pantavisor configuration specifies which revision to boot after it’s been verified by the u-bootloader during the secure boot process. If a revision is in a bad state and fails, the boot and verification process starts over before it rolls back to a good state.


Secure Boot with Pantavisor

Secure boot on a Pantavisor-enabled device proceeds much as previously described. But the main difference is that the bootloader verifies both the kernel and Pantavisor. Pantavisor then is the first executable run by the kernel (the initrd process).  When secure boot is enabled on startup, Pantavisor initially verifies the signature of the ‘state configuration file’ through the PVS (or Pantavisor signatures) which canonically describes the complete state of the containerized system. PVS signatures use JSON signatures to sign all or parts of the state configuration files. Pantavisor then validates them with a set of trusted CA certificates that are baked into Pantavisor at factory time.

Within the state configuration file there is a section for each of the applications that specifies all of the artifacts associated with the application container. Pantavisor verifies each of the specified objects to confirm that they have not been modified.

Once the validation of the content within the application containers is complete, Pantavisor launches the app container and then securely functions as the trusted minimal container runtime of the system, starting and stopping any specified application containers as well as managing their life cycles.

Final Thoughts

With the entire Linux system containerized, Pantavisor easily implements and enforces the secure boot process. All artifacts are declaratively specified in a signed state configuration file so all of the components of the system can also be cryptographically verified at runtime.

Pantavisor enables developers to implement containers on embedded Linux systems. Take advantage of any standard Linux distro or one of the specialty embedded Linux distros to create your product and then easily manage updates and patches across teams and product lines with containers. Pantavisor’s git-like container management tools lets you save and edit device states so that you can share, deploy and secure fleets over the air from Pantacor Hub.

Try it for yourself with this easy to follow Getting Started Guide.