The past couple of years have seen a push from cloud-centric companies telling the embedded Linux ecosystem and community that we should make do with cloud technologies like Golang, NodeJS, and Docker. However, most are unaware of the challenges of cramming cloud tools onto resource-constrained embedded systems. Even though some of these frameworks have the right intention, for example, LXD containers, most lack the initiative to understand the specific requirements of embedded Linux devices.
At a recent FOSDEM conference held online early this February, Ricardo Mendoza (@ricmm) of Pantacor explores how containers on embedded systems help speed up development cycles by enabling modular software architectures. He also spoke about the fundamental requirements of embedded Linux and how modern container technology written for embedded can meet the specific needs of that world.
Docker is too resource-heavy for most embedded boards.
Cloud developers have advocated running Docker on embedded devices. But the problem is that most embedded Linux boards do not have the resources to support a Docker engine. Of course, some large spec devices like Raspberry Pis definitely support resource-intensive managers like Docker. Still, those types of devices running today are not the norm. The reality is that many gadgets, smart home automation devices, and other embedded Linux IoT technology are low spec.
What is the solution? Do we give up on trying to run containers on low spec boards?
Not entirely, since containers offer many well-known benefits:
- Efficient resource utilization – Less overhead and resource usage.
- Modularization – Better isolation and the separation of dependencies.
- Portability – Ability to reuse code and software components across projects.
- Improved security
- Ability to implement automated DevOps practices such as CICD pipelines
Datacenter vs. Embedded Devices
Docker containers and OCI containers make sense in the cloud, where resources are near limitless. However they don’t work with the majority of embedded Linux devices today.
In a datacenter running on Kubernetes, infrastructure is now a lot more resilient than in the early days of the web. In the case of Kubernetes, when a cluster fails or runs out of memory, the system can self-heal by horizontally scaling and adding more memory when an application needs it. As a result, there are near-infinite resources, which is the opposite of embedded.
As mentioned, embedded devices are often running on limited and constrained resources. In addition to this, according to Ricardo, these devices are often performing mission-critical duties. Because of this, they require extreme resilience. Your router, for example, can’t spin itself back up if it gets corrupted during an update. If you live in a northern climate with winter, it would be a disaster if your thermostat randomly fails. Interruptions in service for embedded devices can also require an expensive visit from a technician which is to be avoided at all costs. While failure or downtime in the cloud is also a disaster, unlike the cloud, most embedded devices don’t have the resources or the capability to spin themselves back up again.
Why containers for embedded?
You will likely use Yocto, Buildroot, your own distro, and maybe even OpenWRT to build your embedded system. But most importantly once it’s built, you’ll need a way to manage the system efficiently, which is how containers can help.
“We need to change the way we are updating embedded devices. We are seeing the same issues in embedded as in the early cloud days, where applications and software, in general, were just getting too complex to manage,” says Ricardo.
Firmware is monolithic and complex to update
When you look at embedded devices running for a long time in the field, apps can grow into an unmanageable monolith.
And to make matters worse, embedded development practices that haven’t changed in 25 years can lead to:
- Long release cycles prevent fast innovation.
- Single, giant monolithic releases that can be error-prone and also time-consuming to produce.
- Stale, outdated devices that can be insecure and vulnerable to attack.
- An inability to add new features fast enough.
In an ideal world, we need to gradually and in a controlled manner replace the legacy monolithic firmware and applications with a modular approach.
The easiest way to modularize a system is through containers. With containers, you can divide your application and firmware into self-contained components and, if done right, also manage the dependencies independently. This enables developers to deploy updates and new features in modules. A modular system also allows developers to retain the functions of a legacy application while gradually phasing it out over time, and eventually replacing it with a containerized and modular approach.
Dockerd is not built for embedded.
Most of the OCI-compliant runtimes and the dockerd container manager were not built with embedded in mind. They were not optimized for resource-constrained devices and, as a result, cannot run on these types of embedded Linux devices.
A minimal container runtime built for embedded
Pantavisor proposes a slightly different architecture that doesn’t rely on a huge OS running on top of a container runtime with an additional layer of container applications. Instead, Pantacor takes that paradigm and shifts it slightly.We call Pantavisor the minimal container runtime.
A legacy architecture combines a large Host OS with a Docker engine sitting on top before you even get to your business logic. This type of architecture is way too much overhead for most low-spec devices, says Ricardo.
How much Host OS do you really need in an actual embedded use case? Can you, in fact, modularize and create existing functions in software so that they are a part of the particular device?
Pantavisor does away with the host OS layer and instead containerizes it. A minimal container runtime can set up one to many userland relationships using containers and other tools. In the diagram above, you can see how the host OS runs in a container in the userland. With Pantavisor acting as the minimal runtime container manager, the host OS is seen as just another application that can run critical underlying services which is quite powerful. For example, a containerized host OS can provide critical services like networking or maybe even power control (for example, on a fridge) that can be updated with the rest of your applications.
init_user_ns and boot time
The kernel first runs the init_user_ns (init_user namespace), which is created at boot time. The init_user_ns is a user namespace that belongs to the underlying host system or the root namespace. Because namespaces in Linux are hierarchical, a process in the init_user namespace extends its capabilities to all others since they are all descendants of the root namespace.
By running a minimal container runtime in the init_user_ns , all containers, including the one with the host OS, are able to run low level OS type software functions in the userland.
How does running a container manager as the init user namespace help?
With a container runtime at the kernel layer, you gain several benefits. The most significant benefit is the ability to control and manage the entire lifecycle of both the firmware and software with modern DevOps and automated workflows. It allows you to decouple the firmware from the higher-level software running on the device so that you can manage the lifecycle of those components through a standard protocol.
A minimal container runtime should be a baseline piece of infrastructure. It is much closer to the kernel’s role than it is to the userland itself. A minimal container runtime needs to fit on all types of devices and resources and be as portable as possible. Unfortunately, says Ricardo, portability is challenging to achieve with Node.js, or Golang. Because LXC containers are written in pure C, and was the first true container runtime, it makes more sense to use C for this purpose.
Pantavisor – a nimble and lightweight container manager
At Pantacor, we built a lightweight container runtime and manager in C.
Tiny Footprint
Built for embedded Linux devices at 1MB with strict build rules. Small NAND/NOR flash devices to Raspberry Pis.
Single-purpose system
Pantavisor doesn’t try to be a full userland. Its only purpose is to orchestrate the lifecycle management of your device’s userland, kernel, firmware, and app containers.
Highly portable
Built-in C using a pluggable runtime architecture with LXC containers. Supports any architecture.
Lightning fast
No performance hit with all the benefits of modern lifecycle management.
View the rest of this talk:
Final Thoughts
Pantavisor is a framework for building and managing containerized embedded linux systems for IoT. Try it out for yourself with one of the step by step guides.
Download the whitepaper for more information on how Pantavisor containerizes embedded Linux systems:
