Embedded Linux is changing from making innovative apps for low-cost, low-spec devices to one where powerful hardware runs more complex applications. While resource-intensive devices will become the norm, low-spec devices will still be the ones delivering the volume and the backbone of the consumer industry in today’s embedded Linux Internet of Things (IoT) ecosystem. With the explosion of the connected IoT on the intelligent edge, it’s more important than ever to keep devices up to date and secure. We discuss the challenges faced by embedded engineers on managing firmware and their apps on low-spec embedded devices. Finally, we’ll describe how containers and other cloud-native technologies can help automate and make IoT Linux distros secure and portable.
Three challenges in managing low-spec embedded devices
#1. Keeping embedded systems lean across diverse hardware
Most embedded devices for IoT are single-function and single-purpose and fitted with minimal hardware capabilities that support their intended purpose. In addition, the diverse set of hardware can have limited flash memory with a minimum of 32 MB of NAND, NOR, or EMMC storage with a minimum RAM of 64 MB. These constraints, as well as the diversity of hardware, can limit its processing and networking capability.
Because of these limitations, any firmware and apps created for the appliance need to be efficient and lean without unnecessary binaries, libraries, and modules.
#2. Lack of portability and automation
Most low-spec single-purpose devices aren’t designed to be updated on a continuous basis. A big issue is managing the device state and having the ability to roll back in real-time in the case of a catastrophe. For example, a simple power outage or a bad patch can turn the device into a brick if you do not have a failsafe method of updating and returning to a good state during the update.
Modern cloud-native tools and practices are not common among embedded developers. But now, with the proliferation of IoT apps and the need to apply Linux critical security patches as soon as announced, release teams must embrace DevOps methods to automate and manage firmware and software upgrades to appliances more efficiently.
#3. Customizing the Linux distro
A typical vertically integrated vendor-branded Linux distro includes many redundant modules and packages that are overkill for single-purpose devices with limited resources. Developers instead need to customize their distros to their use case.
But even after specializing your embedded system with custom kernel modules and libraries, most need a way to keep a fleet of thousands of devices’ firmware and software current and secure. Unfortunately, the IoT distros widely in use today are not easily customized and secured without investing significant time to learn the quirks of a particular distro and its set of tools.
Popular Linux distros for embedded computing
Developers working in embedded need a distro that is easy to use and simple to customize for their particular hardware.
The following are some of the best-known Linux distros for embedded computing. However, none of these distros have a standardized “ready-for-product updates” method available. If that’s what’s required, you will need to roll your own.
1. Yocto: The Yocto Project is a popular Linux distro for embedded and it builds on the OpenEmbedded project. Yocto refers to themselves as a ‘meta-distro’ that offers a customizable Distro suited to an embedded device’s hardware constraints. With layers that can be added or removed new features and functionality can be added or removed. Yocto supports a wide range of architectures and also releases updates on a regular schedule. But unfortunately, Yocto can be challenging to work with due to its steep learning curve.
2. Buildroot: Like Yocto, Buildroot is also a distro that allows you build a customized root file system for your embedded device. Buildroot is a leaner version of Yocto but has the limitation that you cannot install updates to a running system. Instead, you’ll need to rebuild the firmware each time before deploying it to the device. In addition, because the framework is so lean and minimal, it can make customization difficult. See, Deciding Between Yocto and Buildroot for an in-depth discussion on the differences between these two distros.
3. Open WRT/LEDE: is probably the most popular framework for embedded devices. It creates custom binaries that are network-accessible. The distro is specific to consumer router firmware and is also very good for general network-focused embedded designs. It features a writeable root file system instead of firmware that is based on read-only file systems and that don’t allow modifications without rebuilding and flashing a new image. It supports a wide variety of architectures and allows you to build distros specifically for each.
4. Raspbian: It is impossible to talk about IoT devices without bringing up the increasingly popular Raspberry Pi. Raspbian is an open-source OS for the Raspberry Pi built on the Debian UNIX OS. Raspbian has versatile uses, from powering low-spec sensors to building more advanced IoT systems, and yet still makes its way into consumer hardware. A drawback to this distro is that it doesn’t fully support any other architecture besides variants of Armv7 and is not easily mixed with standard Debian packages.
5. BalenaOS: runs on top of the Yocto project and can also run Docker containers on your IoT devices. While the OS is containerized and updates are simplified through an external server, you lack complete control over how your distro is specialized as a developer. With this lack of control you may end up with excessive and unnecessary libraries and modules. It also means you are dependent on a single vendor for critical OS updates.
6. Android : Android is a popular distro for touch screen-based device experiences with an app model.It is an operating system based on a modified version of the Linux kernel and other open source software. One big issue with Android is that it is a pure “dropped-over-the-wall-based” ecosystem where no one can easily influence and have input on how upstream code moves forward.
Managing Linux distros on IoT
Unlike cloud systems, embedded systems are more vulnerable to faulty patches or a loss of power during an update, which can quickly turn an embedded device into a brick or cause it to load into an unstable state. Very often, the only recourse for an organization is to send an engineer to manually fix the issue or recall the device. However, a more stable option is found in containers and their portability and ability to apply automation while updating device fleets.
Why containers are the best solution for low-spec devices at the intelligent edge
There have been multiple attempts to use traditional distros like Ubuntu Core and Yocto as a place where 3rd-parties can contribute software at the package granularity with the hope that there would be a way to maintain the high quality of a fully integrated docker-like, distro repository and also gain the agility and innovativeness of a thriving 3rd-party ecosystem. But it has been seen that using “packages” or “layers” is not the right level of granularity to make such a collaboration ecosystem work.
With Linux containers, however, it’s now possible to keep the granularity at a level where 3rd parties can deliver components as fully working units that – if done right – will only have very minimal interdependencies between each other. In the cloud, only the protocols serve as the interdependencies of the modules, and this allows for better abstraction and easier combination of 3rd party components and ultimately a more complete end-user experience
Containers are lightweight, virtualized software environments that run isolated instances of an application or an OS on a host machine, allocating the required resources. Containerizing the distro makes its components modular. These modules can then serve as building blocks and can be used to build specialized distros that are deployed to fleets of devices. Containers revolutionize firmware updates through portability and automation allowing for updates and critical patches to be applied or rolled back without taking down the entire system.
Linux containers (LXC) hold the answer
Lightweight Linux containers (LXC) are best suited for containerizing system-level components of the OS. In addition, LXC containers allow you to safely update individual components of a distro in isolation without risking a complete system failure. They accomplish this by creating partitions in the device and running instances of the Linux OS in each partition – one instance runs the old software and a second instance installs the new software. In this way, if a buggy update is installed or if the power cuts out halfway through an update, the system can easily roll back to the previous version of the software in the first instance without compromising the functionality of the embedded device or turning it into a brick. For more information on LXC containers, read our previous blog post on LXC vs. Docker.
Final Thoughts
Companies looking to upgrade their IoT infrastructure must design embedded systems to handle modern software delivery methods like DevOps. Containers are critical to this transition and ensure that enterprises can roll out updates to keep their IoT devices secure against attack and up to date with cutting-edge developments without risking a costly failure or, worse, a security breach.
Pantavisor Linux is an open-source framework for building containerized embedded Linux systems. Pantavisor leverages LXC containers to make the IoT operating system more modular by virtualizing the distro. Pantavisor offers developers a simple way to build specialized Linux distros, which can be deployed consistently across fleets of IoT devices.