Many developers working with cloud technologies like containers see the Internet of things and embedded Linux devices as an extension to the environments that they work in today. But the reality is that developing and maintaining applications for embedded Linux devices comes with a unique set of challenges and workflows. Some embedded engineers who create new IoT products have not embraced modern tools like containers and their practices for a variety of reasons.
In Part 1 of this two-part post, we’ll discuss typical embedded Linux build workflows and processes that engineers use today. We’ll then describe how containers can improve developer velocity and time to market and how containers can more easily keep devices up to date and secure.
Part 2, we’ll take you through a hands-on tutorial on building containerized embedded Linux systems with our open source project Pantavisor. We’ll also show you how simple it is to deploy over-the-air (OTA) updates to your IoT fleets with Pantacor Hub, our OSS SaaS dashboard, and device state repository.
Software development is more complex
There is no doubt that software development, in particular cloud computing, has become a lot more complex over the years. This same level of complexity also applies to IoT and embedded Linux systems. Embedded devices are increasingly more sophisticated and will continue to get more so. The current Raspberry Pi and Nvidia boards are cases in point. These are examples of a system on a chip (SOC) that provides many resources with many innovative features that can make developing and managing apps and other systems a lot more complicated.
The range of skills needed for cloud developers is broad and varied, both from an individual developer’s point of view and across an entire team. Cloud developers need to wrangle many different systems, from source control repositories like Git to automated continuous integration and deployment pipelines and orchestration systems like Kubernetes. Notwithstanding are the languages and frameworks, and storage systems you also need to master to develop your app before you even begin deploying it to the cloud.
Embedded engineers and hardware
Embedded engineers have the added requirement of managing the hardware. Unlike the cloud where there are near limitless resources, embedded engineers must also evaluate the board’s capabilities and limitations.
Embedded Linux products also tend to be in the marketplace for a long time, sometimes up to 10 years. And so, an additional consideration is that the software and firmware must be reliable and easy to update. Most devices are connected, and like bare metal servers are fundamental components of IoT infrastructure. The last thing an embedded systems engineer wants to worry about is a critical software bug that brings down an entire network or ecosystem.
Embedded engineering and modern cloud technologies
Many of these embedded-specific requirements can be met by adopting modern cloud technologies like containers and their practices. But there are also many other benefits of adopting containers that can help embedded engineering teams, such as a faster turnaround time for proof of concept prototypes to production-ready and an increase in development velocity.
From proof of concept to production ready
When dealing with new and complex frameworks and libraries, embedded engineers must stay productive and ahead of the curve. For example, engineers need to quickly build and iterate on a proof of concept or a product prototype that can be assessed and tested with customers to quickly decide whether it’s a viable product to build and take to market.
Speed to market
With the prototype tested and approved, you need to be ready and get it into the market as soon as possible. If the test is successful, you will also want to be the first on the market with this new generation of products that are hopefully ahead of your competition.
Long time horizons for lifecycle management
Embedded Linux devices require maintenance over a long period and what’s needed is a reliable way to update and maintain the device securely. For embedded systems, you need a consistent power supply and an easy way to roll back or forward to a good state should something go wrong and not end up with a bricked device.
Building embedded Linux systems without containers
Building embedded Linux systems have changed little over the years. Although there are distros specific to embedded systems like Buildroot and Yocto, the development methods and practices surrounding the design and architecture of embedded systems remain similar to those 20 years ago.
Traditionally, embedded teams have slow release cycles consisting of monolith releases every few months or even every year that contain every small change into a single large release. This type of release may work for closed devices. But what if your devices are connected to the internet or in a fleet? If this is the case, some of these changes and updates will need to be delivered quickly.
Single image monolithic deployments
Below is a traditional workflow embedded Linux engineers will embark on when creating an embedded system. Once compiled and debugged, the result is a single image that gets deployed to the board.
After you assemble and customize the board support packages and the Linux kernel libraries and modules, you need to decide on the frameworks and libraries to use for your app. The entire system then gets built in layers or “recipes” with Yocto or Buildroot.
Of course, this is never as simple as it sounds, and often when cross-compiling with a specific toolchain, there are dependency conflicts to debug. However, once completed, the final result is a single monolithic image deployed to the device.
Pros and cons of a single image deployment
An advantage of a single image is that it can be optimized to run much faster on the device. A con, though, is that if you have a critical patch, a new feature, or any other type of update, you must repeat the whole process, which can be time-consuming.
Containers help manage complexity
Remember the days when developers said, ‘works on my machine!” With containers, you can now make that happen. Containers, in essence, allow you to move your machine with an app and its underlying dependencies onto a different machine and be confident that it runs as intended. This is because containers package an app and any dependencies as standardized units that can run in any environment, including embedded Linux devices.
Packaging up all dependencies into a self-contained unit enables systems and apps to do their jobs without any outside help and offers many unique advantages not available in monolithic architectures.
Portable building blocks for embedded Linux systems
A significant advantage of self-contained standardized software units is the ability to divide your monolithic system into logical components. Each component can stand independently with an interface between containers and a container runtime to manage them. You could, for example, separate the system-level components from the applications. You can also use any language or framework you want in the userland for your app without worrying about what OS is running underneath. See, “Mastering Containers on Embedded Linux Devices” or watch the talk “Is the (Distro) still Relevant” for even more information.
Portable building blocks offer the embedded world the same benefits of language independence that cloud developers work in today. Today, most cloud developers do not even think about what Distro they are running or even how many other languages are running in a cluster. Developers instead spend their time on their app or service to provide value for customers rather than the infrastructure. These same benefits can also extend to embedded Linux engineering teams who are building the Internet of Things.
More efficient and agile engineering teams
Another advantage to componentizing kernel libraries separately from the userland is the ability to hand off the higher-level app development to another team. For example, the embedded Linux OS can be componentized by the root filesystem, Bootloader, the kernel, and the BSP in separate containers and other optional utilities, like networking and graphic libraries in containers. Once the system is built for specific boards by the hardware engineers, standard system-level containers can be shared across projects with a simple Docker compose file.
(From: Martin Fowler’s blog- WaterfallProcess)
Likewise, frameworks used by app developers, for example, UI or distributed storage, can also be built as containers and shared in a portable way between app development teams. Again, this makes for more efficient development and assists with quickly turning out a Proof of Concept and then ultimately getting a production-ready product to market much more rapidly.
Simplified software lifecycle management
Another benefit of containerized embedded Linux systems and app libraries is they can be updated more quickly and efficiently. Instead of starting over and building a new monolith with your update, with containers’ ability to manage different versions of libraries, you can update individual components without rebuilding the entire system each time. Because containers are immutable and portable, you can also automate your test and deployment as CICD pipelines, choosing from a myriad of open tools available to you and quickly build new containers with updates and security patches. All of this ultimately speeds up product delivery and increases IoT security.
At the end of the day, implementing DevOps in your embedded engineering workflows provides the ability to quickly and efficiently update your IoT devices. DevOps, containers and automation is what’s going to keep IoT fleets more secure while at the same time continuously offering new services and functionality to your customers.
Pantavisor – a framework for building containerized embedded Linux systems
Pantavisor allows developers to implement containerized embedded Linux systems. Take advantage of any standard Linux distro or one of the specialty embedded distros to create your product and then easily manage it across teams and product lines with containers. Pantavisor’s git-like container management tools let you save and edit device states so that you can share, deploy and secure fleets over the air from Pantacor Hub.
Final Thoughts
This post described how embedded systems are traditionally built, compiled, and updated as a single monolith. We then discussed how containers, in particular lightweight containers, can benefit embedded teams and system engineers who need to update and keep devices secure. The following post will take you through an example that shows you how to containerize your system and how simple it is to maintain it.
Visit Pantavisor.io and try out a tutorial to see how it works.
Questions or Comments?
Ask a question on our Community forum or submit an issue to the Pantavisor GitHub repo. We’d be very happy to hear from you.