Many IoT and other embedded products are primarily built with Linux. The advantages of using Linux for your embedded projects are apparent, such as a large community and flexibility and the ability to contribute that can help drive the direction of the product features.
Despite the many benefits of Linux, it’s still challenging to promptly update embedded devices in the field. As mentioned in other posts, the distro itself needs to build more features into the firmware to properly support embedded. Unfortunately, the slow process together with the lack of available tools to maintain firmware can lead to a proliferation of insecure software and systems.
In this post, we’ll first discuss the architectures that can help increase deployment velocity. We’ll then step through a tutorial on how Pantacor’s technology automates kernel and firmware updates in the BSP for IoT fleets to reduce critical vulnerability exposures.
Critical vulnerabilities in IoT are on the rise
Lately, there has been a proliferation of malware aimed at taking over security cameras, routers, and other common household items to create botnets to mine cryptocurrency or launch DDoS attacks. In other cases, attackers can control a device to record keystrokes to steal identities. Any of these scenarios are to be avoided at all costs.
Embedded Linux development requires an active update strategy
Unlike some server and desktop OS’ that automatically push out software updates to users, embedded Linux developers must look for Linux kernel updates independently. This means developers need to be aware of which open source components they are using in their products and then keep track of when new vulnerabilities are discovered so a fix can be applied promptly.
Customizing the Linux stack
For embedded systems, customizations typically occur in the Board Support Packages (BSP) that include the kernel and supporting modules and, of course, when you add your particular app to the userland.
The typical embedded system uses Yocto or Buildroot or others to build the system. The embedded system is compiled, cross-linked, debugged, and then deployed as a monolith onto the device. Updating monolithic systems can be complex, slow, and error-prone when you are required to compile and validate the entire system each time before deploying it.
How Pantavisor helps with system updates
Pantacor takes a monolithic system and turns it into a set of components using containers. Docker is the default container implementation in the cloud, with Kubernetes as the container orchestrator. With embedded, however, hardware resources are more limited, and so are the operational requirements and challenges, which are vastly different from what a cloud server needs.
Any resources consumed by a container manager removes them from the core functions and apps running on the device. On resource-constrained devices, standard cloud technologies like dockerd are too resource-intensive. Instead, embedded requires lightweight containers with an optimized embedded-first container manager to preserve space.
To get the benefits of container technology without increasing device resource requirements, Pantacor implements native Linux LXC containers and an optimized embedded-first container manager that works with the Linux kernel.
Unlike dockerd, which runs in the typical distribution userland, our embedded-first container manager, Pantavisor, operates lower in the stack between the kernel and the userland. This type of process allows for both your apps in the userland and the host OS to run in containers, giving you complete control over the lifecycle of the entire embedded system.
Updating the BSP
The components and binaries in the BSP are not containerized. The bootloader first runs the kernel and then it runs Pantavisor (the container runtime manager) in the root namespace. However, Pantacor componentizes and packages the BSP to make it simple to update. See, BSP in the Pantavisor Docs.
BSP components running in a Pantavisor-enabled system
These are the components found in the BSP for the Raspberry Pi 3 architecture. If you’ve flashed a device with Pantavisor and then claimed the device on Pantacor Hub, you can navigate the BSP files from within Pantacor Hub:
| File | Description |
| build.json | Description of the BSP build as reproduced from git. |
| firmware.squashfs | Board firmware in compressed Linux format. |
| kernel.img | Linux kernel img |
| modules.squashfs | Added Linux modules in compressed Linux format. |
| run.json | Defines the order the bootloader should run each component. Special add-ons can also be specified at initrd instead of at compile time. |
| src.json | The BSP manifest and url for pvr cli. |
| pantavisor | The container runtime binary. |
How BSP components appear in Pantacor Hub
How to update the BSP
Before you begin, you will need to install the following:
- Download and install the PVR command line utilities.
- Sign up for an account at https://hub.pantacor.com
CI pipeline for automated BSP builds
To simplify BSP updates, you can take advantage of the CI pipeline in GitLab to automate the deployments. This pipeline also maps to specific Fleet channels where they can be scheduled and then applied to millions of devices.
Three options for updating BSPs with Pantacor
There are three options for updating the BSPs on your device fleets:
- Use Pantacor’s automated daily builds, create your own Pantacor Fleet channel and manually update your devices from the command line or through the Fleet UI.
- Build the BSP by running the BSP-CI pipeline yourself, and then map it to your custom Fleet channel.
- Build the BSP packages from scratch without using the CI pipeline. This is the most flexible option, but is also the most involved.
Option #1: Updating device fleet BSPs from the daily build
If you don’t need to customize your BSP, you can take advantage of one of the reference BSP packages available on Pantacor Hub, that are updated and built daily. The Pantacor team runs a CI pipeline in GitLab and builds the BSP for supported architectures every night. You can opt to set up a fleet channel yourself or you can also deliver the updates to the existing pre-configured Fleet channels.
Note:The Pantacor engineering team occasionally baselines against the Linux repo and also fixes bugs. However, updating the Linux kernel and other modules and drivers for your own device fleets will generally be a task for your own development team. However, we do provide the tools and the infrastructure to make your BSP updates simple.
For example, to update your Raspberry Pi 3 devices, you can run the following:
pvr merge https://pvr.pantahub.com/pantahub-ci/arm_rpi3_bsp_latest#bsp
pvr checkout
Where,
- [https://pvr.pantahub.com/pantahub-ci/arm_rpi3_bsp_latest#bsp] is the cloned URL from a specific device architecture found at https://hub.pantacor.com/u/pantahub-ci/devices
For other architectures besides Raspberry Pi, clone the architecture’s URL from the list of latest BSP builds at: https://hub.pantacor.com/u/pantahub-ci/devices and run the above command.
Click on one of the architectures and clone the URL:
Next, define a channel in Pantacor Fleet
Fleet is an add-on to Pantacor Hub that enables you to group, and update devices based on meta-data settings and even rules to select devices from your fleet. Updates can then be rolled out right away or you can schedule an update release for specific times or based on events.
Once you’ve cloned the URL for your architecture from the daily build:
- Launch Fleet from http://fleet.pantacor.com. Note that you must have signed up for Pantacor Hub first. You will then be automatically authenticated on the Fleet API with your hub credentials.
- Create a new channel by selecting Channels —> Add New.
You’ll need to fill in the following fields:
- Name – channel name must be unique
- Description – what’s the purpose of this channel?
- Channel type – Merge source to device or clone source to device
- Merge overwrites any other states or files with the update
- Clone adds the new source to any other source state files already on the device.
- Scheduler Type – either manually or automatically. Manually means you will have to push the button yourself. Automatically will publish the update immediately after the channel is updated.
- Start release offset – optional. If desired add an offset time before the release gets rolled out.
- Default release duration – optional. You can add a total time commitment for the release to complete.
- Source PVR URL – Add the cloned URL from the daily builds here.
Note:This field also supports URLs from multiple sources. These are assembled on the channel from several components and can also keep track of and trigger a new release from each source independently (See, “Firmware customization and deployment strategies for embedded Linux fleets” for more information on this feature.) - Credential – provided by your Pantacor Hub login.
- Required Labels – Labels included in the meta-data that should be included in this channel.
- Selection Labels – Specific meta-data and rules to select the labels to include in this release.
For more information on how to apply rules to meta-data see Creating a channel with the new virtual device.
Option #2: Updating your fleets with the BSP CI pipeline
Configure the PV manifest
Before running the BSP CI pipeline, you will first need to define the manifest for Pantavisor. To do that you can clone the repository and make your changes:
git clone --recursive https://gitlab.com/<group-name or username>/pv-manifest.git
You can choose to build automatically, either pulling a Pantavisor image from a tag or scheduling the release update to trigger a new job and build a new image using the latest tag. Alternatively, you can build a new release from scratch. See the topic: Pantavisor Build System for up-to-date information.
Running the BSP-CI template
Once you’ve defined the PV manifest and updated the BSP components, you’re ready to run the BSP – CI pipeline to build the binaries, and a flashable image that gets delivered to a channel in Fleet.
This template needs to be imported from a GitLab project with a release.xml and default.xml manifests generated by repo:
include: project: 'pantacor/ci/device-ci' ref: '011' file: '/yml/bsp-ci.yml'
For up to date information on how to do that see the following references: Pantavisor CI and BSP CI Reference
Option #3: Building the entire system from your own machine
This third option is for those embedded systems engineers who want full control over the BSP components and how thy are built. The Pantavisor build system allows you to build any combination of components for the BSP. The custom build option also enables you to include any revisions(device state information for rolling forward and backward).
Final Thoughts
In this post, we described how Pantavisor and Pantacor technology enables modular, iterative updates. We then showed you how to do it with a hands-on tutorial.
Join our community for updates on new Pantavisor releases and features.
To learn more on how Pantavisor containerizes embedded Linux systems, download our free whitepaper:
