Mastering Containers on Embedded Linux
Despite what’s been reported by some, embedded devices and the Internet of Things are already pervasive, but many are running out of date and insecure firmware and software. As new hardware architectures continue to release, these challenges will become even more common. Keeping IoT systems secure requires a standardized method for updating the firmware and software. One way to achieve this is with open source technology like containers to implement modern software development practices like DevOps that can easily manage embedded Linux fleets.
Table of Contents
Containers on embedded Linux
In cloud computing, containers have been instrumental in how developers build and automate application deployments. For cloud developers, containers and the orchestrator Kubernetes have essentially added a layer of abstraction to the underlying infrastructure. Many developers today don’t need to think about the OS they are running on and instead they focus on delivering features. Because they don’t need to think about a particular OS or distro, they can deliver value using whatever language they are most comfortable with. In the end, this results in more features deployed to production and into the hands of customers.
Deploying containers to embedded devices is not completely new. There are several companies today, which we’ll discuss later on in this article, who implement containers on embedded devices to manage both systems and applications in the userland. But as we’ll learn, most container implementations on embedded systems are still dependent on the underlying Linux distribution and most also rely on a resource-intensive Docker Engine to run directly on the device.
Docker, VMs vs Linux containers on embedded systems
Before discussing the details of container implementation on embedded Linux, let’s define the differences between containers versus the older virtualization strategy of VMs and discuss how containers can efficiently run on embedded devices.
Container images are standalone executable packages that include everything needed to run an application: the code, runtime, the system libraries, tools and configuration settings. VMs on the other hand enable users to install and run guest operating systems (OS) over an existing host machine OS. But running systems on VMs come with a fairly high overhead that can result in an over-allocation of resources to an OS running on them. Containers on the other hand solve this problem with self-contained OS environments that package an application’s code and all of its dependencies so that applications run side-by-side in isolation on any host machine and only allocate resources only when required.
On embedded systems like Linux, lightweight containers can be used to isolate the firmware, and any apps running in the userland. And one of the advantages with containers on embedded linux systems is that more than one revision or version of a particular library can be kept inside the container. This solves dependencies issues and also reduces the complexity of building distros for your embedded devices where dependencies can be packaged and easily revisioned, managed and maintained through automated CICD pipelines and other modern software practices.
Benefits of working with containers in embedded Linux development
Efficient allocation and use of resources is one of the most important features for embedded devices where resources are minimal. Containers are well-suited to embedded devices because of their ability to allocate resources only when they are needed.
Other notable benefits for implementing containers on embedded developers include the following.
Runs on resource constrained environments
When implemented correctly and compared with the previous generation of virtualization, containers when implemented correctly are much better suited to run on resource constrained devices.
Consistent development environments
Developing with containers offers many benefits to developers. One of the biggest is the ability to define a set of tools, services that can be kept in source control or in the case of containers in a container Registry that is easily reproduced or iterated on and updated when needed. In addition to this, because of containers’ ability to maintain two versions of a library in one container, you also avoid the complex task of rebuilding a monolithic Linux distro every time there is an update.
Modular and reusable components
Because containers allow you to wrap up all of your libraries, utilities and any other necessary dependencies together into one, it’s simple to compile once and distribute the same or a minor variation of the same toolchain to different boards.
Automated CICD pipelines
Easily implement automated pipelines that consistently run tests, rollback and roll forward and build images and quickly deploy new changes, updates or patches to your embedded Linux IoT fleets. This not only makes for less errors during deployments, but also speeds up your development and enables your team to focus on building out new features rather than fixing infrastructure.
For embedded systems where the resources and even network connectivity can be scarce, the ability to partition and stage updates to devices without bricking them is much easier with automated, and fully controllable CICD pipelines.
Better security to meet compliance regulations
With automation in place, teams can more easily update and deploy CVEs and other critical patches to IoT fleets. This eliminates most of the problems around stale and out of date software and firmware out in the wild. Instead, you can keep your fleets secure and also more easily meet compliance regulations for IoT.
See: UK Introduces New Cybersecurity Legislation for IoT Devices
Distributed microservices architectures
When you adopt containers, it’s easy to implement and take a microservices approach to IoT development. For example, you could build firmware once, containerize it and then customize a library for whatever board you are developing on.
For example, to speed up development, the following can be containerized, componentized and possibly reused in other builds across product lines:
Userland
Root filesystem
BSP: Linux kernel and modules
Bootloader
Breaking up your embedded system into containerized components helps isolate your applications from any internal dependencies. As an example, separating a 3rd party library allows that library to be independently updated without breaking your running application. Along the same vein, any of your open source libraries can also be independently containerized and updated without breaking the rest of the system.
Challenges implementing containers in embedded Linux
While it’s very easy to package up a service into a container and deploy it to a Kubernetes cluster, packaging one up for an embedded system presents several unexpected challenges.
Containers can be resource intensive
If you’re running a Docker engine on the embedded device, there is a very high overhead involved. Most devices on the low end of the spectrum would not be able to run a standard Docker engine as well as multiple containers.
The storage requirements for containers is another consideration that needs to be planned for so as not to diminish the device’s running performance. In addition, some embedded Linux devices have limitations on the number of writes to flash memory. These memory limitations also need to be taken into consideration when running containers.
Cross-compiling binaries for different architectures not simple
Even though containers can manage dependencies and isolate applications, optimizing the binaries with the required libraries for your host system in order to build them for your target device is not trivial. Often you will need a distro specifically built for embedded systems like Buildroot or Yocto. But if you’re using containers, you typically want to avoid the steep learning curve that comes along with these distros. This complex build process is what you want to avoid by using containers in the first place.
Provisioning of the final image
Another challenge specific to embedded systems is being able to deploy a change without needing to cross-compile your system each time. Also, how do you manage the base image? The latter is taken care of with a multi-part automated pipeline, where containers can be built and tested in stages. As for the need to cross-compile a change, the use of containers and their ability to maintain different library versions and their dependencies solves that problem for you.
Software license compliance
Often a single container will host many different open source projects. To meet compliance regulations, you need to manage licenses for those in the container image that is distributed. For containers you also need a full inventory of the contents of all container layers within the image.
Yocto already builds this in for example. But for containers, there is not a built in solution for this problem.
See Also:
Software and Firmware Lifecycle Management at Scale
Comparing container solutions for embedded Linux systems
As mentioned, there are a few companies out there who are implementing containers in embedded systems. Each of the companies below though have a slightly different approach to the problem.
Discussion on differences between containers on embedded Linux systems
Open Source | Docker Engine | Containerized Distro | Transactional Updates | Management Dashboard | Third Party Integrations | |
Docker | ✔ | ✔ | ✔ | ✔ | ||
BalenaOS | ✔ | ✔ | ✔ | ✔ | ||
Foundries | ✔ | ✔ | ✔ | |||
Mender | ✔ | ✔ | ✔ | ✔ | ||
Torizon | ✔ | ✔ | ✔ | ✔ | ✔ | |
Pantavisor | ✔ | ✔ | ✔ | ✔ | ✔ |
The table above shows some of the major features across the board from vendors in the IoT space that offer container-based solutions for embedded Linux:
Open source
Is the project open source, and if needed, can individuals openly contribute or participate in the product’s direction?
Docker engine
Does the project rely on Docker engine to be running on the device? Out of the projects listed above, Pantavisor is the only one that doesn’t run a full Docker engine. Because other projects do run large Docker engines directly on the device, they are not able to run on devices with limited specifications.
Containerized distro
Are you tied to a particular distro or are developers free to use whatever distro they are most comfortable with? Most of the projects listed above use Yocto as the underlying OS for the embedded system, but only Pantavisor lets you containerize the actual OS.
With some of these vendors, like Balena and Mender, you are dependent on the vendor to release and manage any critical OS updates. Pantavisor, on the other hand, offers more flexibility in OS updates whereby CVEs found in upstream projects can be immediately applied to vulnerable devices using modern automated pipelines without relying on a particular vendor notification.
In addition to this, only Pantavisor provides a choice of distro and tools. You are not dependent on Yocto or Buildroot, and can use whatever distro or combination of distros you wish.
Atomic and transactional updates
In general, all updates are atomic, but tend to be isolated on a per-component basis. In the embedded world, the entire firmware/software stack is updated transactionally; either it all works or nothing works and the system gets bricked. In addition to this, updates may need to occur with devices that are hard to reach and sometimes with intermittent power. Therefore any update mechanism to IoT devices also needs to take these factors into account.
Management dashboard
Deployment dashboard for collaboration and device management. With the exception of Docker, all of the vendors listed above provide a SaaS web UI for device and fleet management.
3rd party integrations
What types of 3rd party apps does the project integrate with? Is it easy to manage IoT device data with other cloud native open source tools like Kubernetes and other platforms available from the public clouds?
How Pantavisor implements containers
Pantavisor is a container framework for embedded Linux systems and applications. Based on Linux (LXC) containers, but implemented with a Docker developer workflow, it turns your firmware and userland into a set of atomically shareable building blocks. Pantavisor converts any Docker container into LXC format and runs it on all of your devices, including those with the smallest resource footprint.
With Linux containers, it’s possible to deliver system components as fully working units. Containers are lightweight, virtualized software environments that run isolated instances of an application or an OS on a host machine, only allocating resources when required. By containerizing the disto, its components become modular reusable components that can be used to build specialized distros and deployed to fleets of devices. Turn firmware, the OS and even the BSP into portable, containerized components that can be shared and deployed transactionally over the air to safely manage the lifecycle of your IoT fleets.
“Containers revolutionize firmware updates through portability and automation allowing for updates and critical patches to be applied or rolled back without taking down or bricking the device.”
Manage transactional updates with PVR and Pantacor Hub
Unlike systems running in the cloud, embedded systems are more vulnerable to faulty patches or a loss of power during an update, which can quickly turn an embedded device into a brick or cause it to load into an unstable state. When this happens, the only recourse for an organization is to send an engineer to manually fix the issue or recall the device. However, a more stable option is found in immutable containers that can be used to automate and modernize CICD deployment pipelines as well as other modern software practices that keep device fleets more secure.
At the core of Pantavisor is its git-like command line utility, PVR that allows for fine-grain control over device state revisions, and their history and management. Combined with Pantacor Hub, our SaaS for managing and deploying state revisions, you can easily apply modern software development practices to update and secure your IoT fleets across an infinite number of architectures.
Want to get up and running? Get started with this easy to follow tutorial.