The Internet of Things comprises a wide range of devices all running different architectures and often configured specifically for a specific task at hand. Although a good chunk of the embedded device world does run on Linux, the addition of customized software and systems to achieve that task can complicate maintenance lifecycles to keep them up to date, secure and reliable. What are the minimal requirements for updating embedded Linux devices? How did product makers run updates in the past and still do even today? Lastly we’ll look at how open source technologies like containers and other modern workflows when implemented correctly can greatly benefit embedded software and firmware lifecycle management.
Challenges in the embedded environment
According to IDC connected devices will generate zettabytes of data each year. That volume is set to triple in the upcoming years where there will be more than 55.7 billion connected devices worldwide by 2025, with 75% of them connected to an IoT platform. With this many connected IoT devices, it’s crucial that updating them is simple, efficient and most importantly secure.
But before diving into what is needed for embedded device updates, it’s worth taking a closer look at what the embedded environment looks like. Unlike cloud computing or web programming, your environment in the embedded world is strictly controlled with a number of limitations.
Equipment is often in hard to reach locations
Typically consumer and other devices that you don’t even think about are in remote and difficult to reach locations. If you think about wifi routers and other hard to get equipment, they can not always be updated in a controlled manner.
Network connectivity can vary
The connectivity might be intermittent, it can be over 4G connection or something even slower than that. And you may or may not have secure connectivity, especially if you’re going through public networks.
Product lifetimes can extend beyond 10 years
Product lifetimes vary widely in the embedded space. Some markets like automotive can keep the same hardware for five to ten years, and for home appliances it can be double that. But in the consumer electronics space, the product lifetime is more like 6 to 12 months, so there’s quite a variety.
Power is not always guaranteed
Power issues have to be dealt with, especially in the case of Consumer Service Providers (CSP) whose customers are not enterprises. A lot of these consumer-type devices run on battery power and even if they’re not, there’s no guarantee the owner of the device won’t unplug the power at any given moment. You’ve got to be able to deal with a device that may have intermittent power.
What gets updated on an embedded Linux system?
When working with embedded systems, developers are often building applications that are fixed-function on resource-constrained embedded Linux devices. By resource constrained, we mean that running applications need to function on a minimum specification which can include NAND/NOR flash class devices with under 64MB of total storage. Nothing like what is seen in the cloud, where storage and capacity are dynamic and readily available at scale and on demand.
As mentioned, most embedded devices are single-purpose designs constructed to perform one thing well. Because of this high level of specialization, a Linux distro is often completely stripped down. When It comes to security and other maintenance updates, these are the base components that can require updating on an embedded Linux device:
| Board Support Package (BSP) | This is often supplied by the System-on-Chip manufacturer you are using and it contains the code base that provides base Linux kernel support for the target board that is used in the device. These are often customizable by developers and will also need updating on occasion. |
| Bootloader | This is the first component to run after the device is powered up. It initializes the board and brings up the Linux kernel. |
| Linux Kernel | This core of Linux distro that includes any specialized modules and specific drivers for the app you want to run. This is generally a component that will require updating on a semi-regular basis. |
| Root filesystem | This is where the essential system libraries and other utilities and scripts are kept that keep the system running. Most of these files will need to be updated and upgraded. |
| System applications and packages | This is where your applications are in the userland. Here you will need to update features and occasionally fix bugs. |
Traditional Embedded Distro Updates
Updating embedded systems often requires a complete rebuild of the OS to secure the device. This can be done manually file by file, package by package or in a more portable way layer by layer, depending on the Linux distro you’re using. Once you’ve added in the fixes and upgrades and then rebuilt the OS and tested it, you’re ready to deploy it back to the device from your Over the Air (OTA) device update system.
A big issue with the `rebuild of the entire OS` method is that these days you may have 1000s of devices, many of which have bespoke configurations and customized distros with select packages and modules or even highly specialized board support packages that stray from the original board design. Updating all of these individually or even in smaller device groups is a time consuming and error prone process.
Embedded Linux distros can help simplify updates but requires a lot of effort
Even though embedded distributions and systems like Yocto and Buildroot simplify the process, customizing and building a monolithic bespoke system requires considerable effort to rebuild and maintain. In addition to this, using packages or layers doesn’t always provide the right level of granularity for a truly collaborative and ultimately portable ecosystem to work. What’s needed instead is a more immutable way that provides more fine grained control over OS components. Preferably a system that doesn’t require you to rebuild a monolithic system every time you need to make a small update.
In addition to this, traditional updates usually follow the path of the last two decades that include A/B models. In this scenario, you have one active partition and a target partition. An update is then always applied in full to the one partition before making it run after full system reboot. This approach works fine for single-purpose devices, but the requirements of the connected world ask for multi-function, always-on, highly-reliable systems. An A/B model does not scale enough to meet these demands.
Updating embedded systems with containers
What if you could imagine your entire OS and system as a modular set of containerized immutable building blocks that you have complete control over? With containers, it’s possible to keep the system granularity at a level where 3rd parties can deliver components as fully working units that – if done right – will only have very minimal interdependencies between each other.
A fully containerized system includes the containers for: the Linux kernel, any middleware packages as well as the Userland. Containerization at this level provides you with a fully modular system with more flexible and independent updates across millions of devices if needed.
Note: Pantavisor doesn’t directly containerize the BSP, but it does package it and compress it in a way that makes it very simple to update, see Pantavisor Architecture and BSP configuration in the Pantavisor technical documentation for more information.
Docker vs lightweight LXC containers
Most Docker containers are too bulky to fit on the majority of IoT devices today. But with Lightweight containers like LXC that are specifically designed for embedded system level containerization, you can more efficiently update and manage your Linux system. This allows you to take advantage of the following benefits:
- Not be dependent on a single vendor – Do away with a reliance on an individual Linux Distro and the learning curve and the distro-specific set of tools that comes along with it.
- Immutable modular components – With a fully modular system, you can more easily control any customizations to the OS and make it portable across many embedded devices in a network. In addition to this, in the case of system provisioning when a system has reduced connectivity, a system that is modular allows for staged updates where the base OS can be provisioned before the application if desired.
- Auditable and reproducible LXC containers for your OS – By combining the DX of Docker containers with the lightness of LXC containers, you gain full control over deployments and their reproducibility in addition to system modularity.
For example combining Docker compose files with LXC containers, OS systems can be deployed and tested across different environments and rolled back or forward in the case of a failure.
- Increased deployment velocity – With portable and immutable components, it is simpler for organizations to implement DevOps best practices in their embedded pipelines. Faster turnaround for features means more value add for product lines.
- Increased security with DevSecOps best practices – An increase in portability and velocity that comes with containers allows for more frequent updates and therefore more secure systems and devices. Patches can be applied across fleets whenever they are needed through automated pipelines.
Pantavisor Linux is designed for embedded Linux
Pantavisor Linux uses containers to build modular containerized embedded systems that are fully customizable and more easily managed and updated. It uses a Docker developer experience on top of Linux containers which means you can leverage Docker Compose files for full audibility and control over your system and application containers. In addition to this, Pantavisor Linux converts and runs any Docker container downloaded from the ecosystem, while still maintaining the convenient Docker root stack for portability. Pantavisor lets you compose a completely reproducible system out of reusable building blocks that can be shared across devices and with the greater community from Pantacor Hub.
Tiny footprint for embedded Linux
Pantavisor has a tiny footprint of 1MB with strict build rules. It can be used in any Linux device, from single-core NAND/NOR flash class devices to Raspberry Pis. If it’s Linux, Pantavisor runs on it.
Single purpose system
Pantavisor does not try to be a full userland which makes for a nimble embedded system. Its only purpose is to orchestrate the lifecycle management of your device’s main userland, kernel, firmware, and any application containers.
Highly portable
Built in C using a pluggable runtime architecture with LXC as the main choice, Pantavisor supports any architecture if the core container features are supported at the Linux kernel level. Currently ARM, ARM64, x86, MIPS and RISC-V are in use.
Lightening fast
No performance hit with all the benefits of modern lifecycle management.
As this is not ring-0 virtualization, but rather userland (ring-3) containerization, there is no runtime performance impact on the system. The containerized userland and applications run equal to their bare metal counterparts.
For more information see: Pantavisor Architecture
Final Thoughts
Pantavisor Linux, provides you with the building blocks for a universal Linux non-OS. Deploy, and atomically manage your containerized embedded OS to millions of devices over the air in a portable and reproducible manner. With system updates simplified and modularized with containerization, your team can focus on adding new features and adding other value to your IoT network.
For more information see https://www.pantavisor.io
Questions or Comments?
Ask a question on our Community forum or submit an issue to the Pantavisor GitHub repo. We’d be very happy to hear from you.