Mastering Containers on Embedded Linux

Despite what’s been reported by some, embedded devices and the Internet of Things are already pervasive, but many are running out of date and insecure firmware and software. As new hardware architectures continue to release, these challenges will become even more common. Keeping IoT systems secure requires a standardized method for updating the firmware and software. One way to achieve this is with open source technology like containers to implement modern software development practices like DevOps that can easily manage embedded Linux fleets.

Table of Contents

Containers on embedded Linux

In cloud computing, containers have been instrumental in how developers build and automate application deployments to the cloud. For cloud developers, containers and the orchestrator Kubernetes have essentially added a layer of abstraction to the underlying infrastructure. Many developers today don’t need to think about the OS they are running on and instead they focus on delivering features. Because they don’t need to think about a particular OS or distro, they can deliver value using whatever language they are most comfortable with and in the end this results in more features deployed to production and into the hands of customers.  

Deploying containers to embedded devices is not completely new. There are several companies today, which we’ll discuss later on in this article, who implement containers on embedded devices to manage both systems and applications in the userland.  But as we’ll learn, most container implementations on embedded systems are still dependent on the underlying Linux distribution and most also rely on a resource-intensive Docker Engine to run directly on the device.

Docker containers, VMs vs containers on embedded systems

Before discussing the details of container implementation on embedded Linux, let’s define the differences between containers versus the older virtualization strategy of VMs and discuss how containers can efficiently run on embedded devices. 

Container images are standalone executable packages that include everything needed to run an application: the code, runtime, the system libraries, tools and configuration settings.  VMs on the other hand enable users to install and run guest operating systems (OS) over an existing host machine OS. But running systems on VMs come with a fairly high overhead that can result in an over-allocation of resources to an OS running on them. Containers on the other hand solve this problem with self-contained OS environments that package an application’s code and all of its dependencies so that applications run side-by-side in isolation on any host machine and only allocate resources only when required. 

On embedded systems like Linux, lightweight containers can be used to isolate the firmware, and any apps running in the userland. And one of the advantages with containers on embedded linux systems is that more than one revision or version of a particular library can be kept inside the container. This solves dependencies issues and also reduces the complexity of building distros for your embedded devices where dependencies can be packaged and easily revisioned, managed and maintained through automated CICD pipelines and other modern software practices.

vs-docker-embeddedlinux

Benefits of working with containers in embedded Linux development

Efficient allocation and use of resources is one of the most important features for embedded devices where resources are minimal. Containers are well-suited to embedded devices because of their ability to allocate resources only when they are needed. 

Other notable benefits for implementing containers on embedded developers include the following. 

Runs on resource constrained environments

When implemented correctly and compared with the previous generation of virtualization, containers are much better suited to run on resource constrained devices. 

Consistent development environments

Developing with containers offers many benefits to developers. One of the biggest is the ability to define a set of tools, services that can be kept in source control or in the case of containers in a Registry that can be easily reproduced and iterated on when needed.  In addition to this because of containers’ ability to maintain two versions of a library in one container, you avoid the complex task of rebuilding a distro every time there is an update. 

Modular and reusable components

Because containers allow you to wrap up all of your libraries, utilities and any other necessary dependencies together into one, it’s simple to compile once and distribute the same or a minor variation of the same toolchain to different boards. 

Automated CICD pipelines

Easily implement automated pipelines that consistently run tests, rollback and roll forward and build images and quickly deploy new changes and updates to your IoT fleets. This not only makes for less errors during deployments, but also speeds up your development and enables your team to focus on building out new features rather than fixing infrastructure. 

For embedded systems where the resources and even network connectivity can be scarce, the ability to partition and stage updates to devices without bricking them is much easier with automated, fully controllable pipelines.

CICD Pipeline Embedded Linux

Better security to meet compliance regulations

With automation in place, teams can more easily update and deploy CVEs and other critical updates to IoT fleets. This eliminates most of the problems around stale and out of date software and firmware out in the wild and instead you can keep your fleets secure and able to more easily meet compliance regulations for IoT

See: UK Introduces New Cybersecurity Legislation for IoT Devices

Distributed microservices architectures

When you adopt containers, it’s easy to implement and take a microservices approach to IoT development.  For example, you could build firmware once, containerize it and then customize a library for whatever board you are developing on. 

For example to speed up development, the following can be containerized, componentized and possibly reused in other builds: 

  • User land

  • Root filesystem

  • BSP: Linux kernel and modules

  • Bootloader

Breaking up your embedded system into containerized components helps to isolate your applications from any internal dependencies. As an example, separating a 3rd party library allows that library to be independently updated without breaking your running application. Along the same vein, any of your open source libraries can also be independently containerized and updated without breaking the rest of the system. 

Challenges implementing containers in embedded Linux​

While it’s very easy to package up a service into a container and deploy it to a Kubernetes cluster, packaging one up for an embedded system presents several unexpected challenges. 

Containers can be resource intensive

If you’re running a Docker engine on the embedded device, there is a very high overhead involved. Most devices on the low end of the spectrum would not be able to run a standard Docker engine as well as multiple containers. 

The storage requirements for containers is another consideration that needs to be planned for so as not to diminish the device’s running performance. In addition, some embedded Linux devices have limitations on the number of writes to flash memory. These memory limitations also need to be taken into consideration when running containers.  

Cross-compiled binaries for different architectures with containers not simple

Even though containers can manage dependencies and isolate applications, optimizing the binaries with the required libraries for your host system in order to build them for your target device is not trivial. Often you will need a distro specifically built for embedded systems like Buildroot or Yocto.  But if you’re using containers, you typically want to avoid the steep learning curve that comes along with these distros and generally is what you want to avoid by using containers in the first place. 

Provisioning of the final image

Another challenge specific to embedded systems is being able to deploy a change without needing to cross-compile your system each time. Also, how do you manage the base image? The latter is taken care of with a multi-part automated pipeline, where containers can be built and tested in stages.  As for the need to cross-compile a change, the use of containers and their ability to maintain different library versions and their dependencies solves that problem for you.   

Software license compliance

Often a single container will host many different open source projects. To meet compliance regulations, you need to manage licenses for those in the container image that is distributed. Needs a full inventory of all container layers within the image. 

Yocto already builds this in for example. But for containers, there is not a solution for this. 

See Also: 

Comparing container solutions for embedded Linux systems

As mentioned, there are a few companies out there who are implementing containers in embedded systems. Each of the companies below though have a slightly different approach to the problem.

Discussion on differences between containers on embedded Linux systems​

Open Source

Docker Engine

Containerized Distro

Transactional Updates

Management Dashboard

Third Party Integrations

Docker

BalenaOS

Foundries

Mender

Torizon

Pantavisor

The table above shows some of the major features across the board from vendors in the IoT space that offer container-based solutions for embedded Linux:  

Open Source

Is the project open source, and if needed, can individuals openly contribute or participate in the product’s direction?

Docker Engine 

Does the project rely on Docker Engine to be running on the device? Out of the projects listed above, Pantavisor is the only one that doesn’t run a full Docker engine. Because other projects do run large Docker engines right on the device, they are not able to run on devices with limited specifications.  

Containerized Distro

Are you tied to a particular distro or are developers free to use whatever distro they are most comfortable with. Most of the projects listed above use Yocto as the underlying OS for the embedded system, but none of them with the exception of Pantavisor, containerize the actual OS.

With some of these vendors, like Balena and Mendor you are dependent on the vendor to release and manage any critical OS updates. Pantavisor on the other hand offers more flexibility in OS updates whereby CVEs found in upstream projects can be immediately found applied to vulnerable devices using modern automated pipelines without relying on a particular vendor notification. 

In addition to this of the solutions provided above, only Pantavisor provides the choice of distro and tools. You are not dependent on Yocto or Buildroot, and can use whatever distro or combination of distros you wish.

Atomic and transactional updates

In general all updates are atomic, but tend to be isolated on a per-component basis. In the embedded world, the entire firmware/software stack is updated transactionally, either it all works or nothing works and the system gets bricked.  In addition to this, updates may need to occur with devices that are hard to reach and sometimes with intermittent power. And so any update mechanism to IoT devices, also needs to take these factors into account.

Management Dashboard

Deployment dashboard for collaboration and device management. With the exception of Docker all of the vendors listed above provide a SaaS web UI for device and fleet management.

3rd party integrations

What types of 3rd party apps does the project integrate with? Is it easy to manage IoT device data with other cloud native open source tools like Kubernetes and other platforms available from the public clouds.

How Pantavisor implements containers

Pantavisor is a container framework for embedded Linux systems and applications. Based on Linux (LXC) containers, but implemented with a Docker developer workflow, it turns your firmware and userland into a set of atomically shareable building blocks. Pantavisor converts any Docker container into LXC format and runs it on all of your devices, including those with the smallest resource footprint. 

With Linux containers, it’s possible to deliver system components as fully working units.  Containers are lightweight, virtualized software environments that run isolated instances of an application or an OS on a host machine, only allocating resources when required. By containerizing the disto, its components become modular reusable components that can be used to build specialized distros and deployed to fleets of devices. Turn firmware, the OS and even the BSP into portable, containerized components that can be shared and deployed transactionally over the air to safely manage the lifecycle of your IoT fleets.

“Containers revolutionize firmware updates through portability and automation allowing for updates and critical patches to be applied or rolled back without taking down or bricking the device.”

Manage atomic updates with PVR and Pantacor Hub

Unlike systems running in the cloud, embedded systems are more vulnerable to faulty patches or a loss of power during an update, which can quickly turn an embedded device into a brick or cause it to load into an unstable state. When this happens, the only recourse for an organization is to send an engineer to manually fix the issue or recall the device. However, a more stable option is found in immutable containers that can be used to automate and modernize CICD deployment pipelines as well as other modern software practices that keep device fleets more secure.

At the core of Pantavisor is its git-like command line utility, PVR that allows for fine-grain control over device state revisions, and their history and management.  Combined with Pantacor Hub, our SaaS for managing and deploying state revisions, you can easily apply modern software development practices to update and secure your IoT fleets across an infinite number of architectures.

Want to get up and running? Get started with this easy to follow tutorial.