Guide to Building Embedded Linux Systems with Containers
Have you ever thought about how to make building embedded Linux systems simpler and how modern technology like containers can help? This article takes you through the internals of embedded Linux for building and managing systems and apps for the Internet of Things (IoT). It also describes how embedded Linux systems are architected, and how modern technology like containers can efficiently manage the lifecycle of firmware and software of embedded Linux systems.
Table of Contents
But before getting into the details of containers and embedded Linux, it’s important to step back and define what we mean when we are talking about embedded devices that make up the Internet of Things (IoT).
What is the IoT?
The Internet of Things or IoT sits at the far end of the edge spectrum. One helpful definition is that an IoT device generally has one foot in the analog world. In other words, it can be thought of as a device that functions partially in the real world rather than operating 100% into the virtual world.
Edge Taxonomy (from: Linux Foundation Edge: State of the Edge Report)
The diagram above illustrates a typical high-level overview of what ‘the edge topology’ looks like. With the exception of the on-prem Data Center Edge, when we speak about IoT, we’re talking about the User Edge on the far left of this diagram, namely the Constrained Device Edge and the Smart Device Edge.
In general, devices at the user edge, including the smart edge device, are mostly single-purpose devices with limited resources and that, as mentioned earlier, often operate partially in the analog world.
Examples of IoT devices
Examples of devices representing the Internet of Things include:
Home routers
Set top boxes and other home automation
Smartphones and tablets
Video and multimedia systems
Robotics
Wearable devices
Data sensors: temperature, weather, humidity, etc.
Smart building infrastructure
Microcontrollers and microprocessors
IoT embedded devices are essentially single board computers with a particular microprocessor designed specifically for embedded applications. IoT boards or hardware can range from an 8-bit microcontroller (MCU) to a 32-bit or 64-bit system-on-chip (SoC). Most microcontrollers and microprocessors in use today are embedded into products, such as automobiles, telephones, and household appliances.
While some embedded systems are sophisticated and complex, many, for example microcontrollers or MCU as opposed to SoC’s like Raspberry Pis, have minimal memory and program length with no operating system. Typical MCU use cases include input and output pins such as switches, relays, solenoids, LEDs, simple liquid-crystal displays, radiofrequency devices, and sensors for data such as temperature, humidity, and light level.
Regardless of whether a board contains an MCU or a SoC, IoT apps run on a variety of operating systems including: Linux, Free RTOS, Windows, and Zephyr.
In the rest of this article, we are concerned with embedded systems that run Linux. It’s safe to say that there are an extremely wide range of embedded microprocessors running on Linux. To gain a better understanding of the scope, they can be split into the following size categories:
Small-sized systems, comprising low-powered CPU with at least 2 MB of read-only member (ROM) and 4 MB of random-access memory (RAM)
Medium-sized systems with about 32 MB of ROM, 64 MB of RAM and a medium-powerd CPU
Large-sized embedded system, with power CPUs and a larger memory footprint
(From: What is Embedded Linux?)
Software and Firmware Lifecycle Management at Scale
Benefits of using Linux
Embedded Linux is by far the most popular because of its reliability, configurability and minimal system requirements. Other well-known benefits include:
Open source and built on standards
Because it’s open-source, contributions from all over the world make Linux more robust and secure as time goes on. In addition, the open-source ecosystem provides many components for standard features, from hardware to network protocols, including multimedia, graphics formats, and secure libraries.
Flexible and allows for code reuse
The flexibility of Linux, combined with consistency in performance, architectural tiers, virtualization, shared libraries, and cloud support, makes Linux distros a popular choice among IoT developers.
Supports a wide range of architectures
Because Linux is built on standards, it’s easy to reuse components across architectures and systems. Linux also supports a wide range of architectures such as ARM, MIPS (Microprocessor without Interlocked Pipeline Stages), RISC (Reduced Instruction Set Computer), and others.
Most widely used OS for IoT development
According to the Eclipse Foundation, the top operating system for IoT in 2020 is Linux. Because of this, you are guaranteed a community that offers support and prompt resolutions to security vulnerabilities.
From: IoT Eclipse Foundation: IoT Developer Survey 2020
Royalty free
This is not the same as free from costs. Of course Linux is not completely free since you will need the development team to implement and build applications for it. However, you will be free from paying vendor fees, especially if you rely on a completely open source and free Linux distribution that is supported, maintained and secured by the community.
Building embedded Linux systems
Before talking about containers and building embedded systems, let’s look at how to build embedded Linux systems today. As mentioned earlier, most embedded devices are built to do a specific task on a usually resource-constrained or low spec device. As a result, most embedded developers will need to strip out the unnecessary libraries and modules and create a custom distribution for their particular device and use case.
Components of an embedded Linux System
The diagram below describes typical components in an embedded system.
Bootloader
Technically resides outside of the system and operates differently from a desktop or server system where the BIOS is the first thing that runs. On an embedded system, the hardware starts the bootloader responsible for basic initialization and executing the kernel. On embedded systems, this can be handled by: GRUB, LILO or Das U-Boot. The latter of which is used exclusively on embedded devices.
Linux kernel
Contains the process and memory management, network stack, device drivers and provides services to any user-space applications. The bootloader loads the kernel into memory and runs it. The kernel looks for the ‘init’ program to run first. The ‘Init’ program is responsible for starting up the other services such as hardware drivers, filesystem drivers, the file system mount, services, and other apps. The kernel is only aware of the init program, and if it doesn’t find it, it will ‘kernel panic’.
The kernel on an embedded system is identical to a kernel in larger systems such as a desktop. The main difference is that an embedded Linux kernel is purpose-built to run on a different CPU architecture. See “The Linux Boot Process” for more information.
Root filesystem
The root filesystem contains the C Libraries and the Linux services/commands, and other necessary scripts. The filesystem is the interface between the kernel and the user-space applications. At init time, the kernel loads each program into memory, and it expects them to be organized into files and directories. This is the root filesystem, and it should be created in advance and be mounted to ‘/’ before the kernel can launch ‘init’.
Userland
Userland is where any custom applications or services reside. The init program is also responsible for loading these into memory. Most embedded systems are single-purpose apps to perform a particular function, so apps on embedded systems tend to be lean and minimal. Running Linux on a target embedded processor requires a minimum of 8MB of RAM with most applications requiring at least 32MB RAM. The actual requirement of RAM can depend on the size of your embedded application. Other than RAM, a minimum of 4MB storage memory is also needed. It can be one of the following types:
NAND or NOR Flash
SD or MMC cards
Board support packages
The board support packages contain the drivers and other software that interfaces with the specific hardware and the operating system you’re running. For Linux systems, each board type will have its own set of BSPs. In some cases, if you need the hardware to do specific functions that aren’t included in the default BSPs, you may need to customize these and write your own drivers to support your particular application.
Tools for building embedded Linux systems
You can take any Linux distro, even a desktop version, and strip it down to include what you need. After you’ve selected the appropriate drivers and libraries and their required versions and dependencies for your particular board, you will then begin the process of integrating and cross-compiling the components to rebuild them for your target device.
Cross toolchain compiler
If you were building a desktop distribution, you would normally use a native ToolChain and it would be built for an x86 system. However, in embedded systems development, because of resource constraints with limited storage and/or memory, it is not possible to use a native toolchain. Also, the target embedded device is much slower than your workstation and you may not need all of the development tools available in a native toolchain on your target. For these reasons, you can take advantage of cross-compiling toolchains that run on your workstation and then generate code for your target or specific board.
Popular Linux distros for embedded Linux computing
Creating your own custom distribution is not for the faint of heart. As a result, many embedded developers work with a distro that has been designed for embedded development.
The following are some of the best-known Linux distros for embedded computing. However, it should be noted that none of these specialty embedded distros have a standardized method for embedded Linux firmware and lifecycle management.
Yocto
The Yocto Project is a popular Linux distro for embedded and it builds on the OpenEmbedded project. Yocto refers to themselves as a ‘meta-distro’ that offers a customizable Distro suited to an embedded device’s hardware constraints. With layers that can be added or removed new features and functionality can be added or removed. Yocto supports a wide range of architectures and also releases updates on a regular schedule. But unfortunately, Yocto can be challenging to work with due to its steep learning curve.
Buildroot
Buildroot allows you to build customized root file systems for your embedded devices. Buildroot is a leaner version of Yocto but has the limitation of not being able to install updates to a running system and instead, requires you to rebuild the firmware each time before deploying it to the device. Also with the framework being so lean and minimal, customization can be difficult.
See also: Deciding Between Yocto and Buildroot
OpenWRT/LEDE
OpenWRT/LEDE is a popular framework for embedded devices that creates network-accessible custom binaries. The distro is specific to consumer router firmware and is also very good for general network-focused embedded designs. It features a writeable root file system instead of firmware that is based on read-only file systems and that don’t allow modifications without rebuilding and flashing a new image. It supports a wide variety of architectures and allows you to build distros specifically for each.
Raspbian
It is impossible to talk about IoT devices without bringing up the increasingly popular Raspberry Pi. Raspbian is an open-source OS for the Raspberry Pi built on the Debian UNIX OS. Raspbian has versatile uses, from powering low-spec sensors to building more advanced IoT systems, and yet still makes its way into consumer hardware. A drawback to this distro is that it doesn’t fully support any other architecture besides variants of Armv7 and is not easily mixed with standard Debian packages.
BalenaOS
BalenaOS runs on top of the Yocto project and can also run Docker containers on your IoT devices. While the OS is containerized and updates are simplified through an external server, you lack complete control over how your distro is specialized as a developer. With this lack of control you may end up with excessive and unnecessary libraries and modules. It also means you are dependent on a single vendor for critical OS updates.
Android
Android is a popular distro for touch screen-based device experiences with an app model.It is an operating system based on a modified version of the Linux kernel and other open source software. An issue with Android is that it is a pure “thrown-over-the-wall-based” ecosystem where no one can easily influence or have input on how upstream code moves forward.
How containers make embedded Linux distros universal
Many projects have attempted to build a place for 3rd parties to contribute packages around traditional distros like Ubuntu Core and Yocto. There was the hope that this could provide a fully integrated distro with a Docker-like experience and gain the agility and innovativeness of a thriving 3rd-party ecosystem. But it soon became apparent that “packages” or “layers” do not have the right level of granularity needed by embedded developers to make this kind of ecosystem really work.
Linux containers allow you to maintain the granularity at a level where 3rd parties can deliver fully working components, and if done right, with minimal interdependencies. If you compare this type of ecosystem with what’s happening in the cloud, protocols such as REST APIs serve as the interdependencies of the modules. It’s these types of protocols that allow for a cleaner abstraction and simpler combination of 3rd party components. And with embedded Linux distros, this type of abstraction layer can also be achieved with containers.
Containers are lightweight, virtualized software environments that run isolated instances of an application or an OS on a host machine that allocate necessary resources. Containerizing a Linux distro makes its components modular. These modules can then serve as building blocks and can be used to build specialized distros that are more easily deployed to fleets of IoT devices.
For more information, also see the article: Mastering Containers on Embedded Linux Devices
Secure and fast IoT device updates
Containers revolutionize firmware and software updates through portability and automation. They allow for updates and critical patches to be applied or rolled back without taking down the entire system or bricking the device.
Unlike cloud systems, embedded Linux systems are more vulnerable to faulty patches or a loss of power during an update, which can quickly turn an embedded device into a brick or cause it to load into an unstable state. Very often, the only recourse for an organization is to send an engineer to manually fix the issue or recall the device. However, a more stable option is found in the portability of containers and their ability to apply automation while updating device fleets.
For more information on how containers simplify embedded Linux system maintenance and security, take a look at Alexander Sack’s recent talk:
Containerized IoT devices with Pantavisor Linux
Pantavisor Linux provides product makers with freedom of choice to choose the distro they are most comfortable with. Pantavisor makes it simple to containerize all of the embedded system components as well as apps in the userland.
With your Linux distribution or custom-made firmware userland in a container, you get the benefits of portable container lifecycle management without replacing your distribution. Turn firmware, the OS, and even the BSP into portable, containerized building blocks that can be shared and deployed transactionally over the air through our SaaS Pantacor Hub. Pantavisor implements a Docker developer experience and workflow to implement and run containers. It can also directly convert Docker containers into LXC format to run on all of your devices, including those with the smallest resource footprint.
Pantavisor system state format: Linux systems made with containerized building blocks
JSON format to define complete system stack
Breakdown of monolithic firmware in blocks
Choice of BSPs with Kernel and Drivers
Mix&Match Containers for OS & Apps
Ability to deploy safe/transactionally built-in
System States can be exported and imported in easy to share format with pvr tool.
For more and information and tutorials visit pantavisor.io