Firmware Customization and Deployment Strategies for Embedded Linux Fleets

In a previous post, we discussed how to build an embedded Linux Board Support Package (BSP) with automated CICD pipelines and then apply those firmware updates to millions of devices. We also dove into configuring release channels and how to schedule updates. In this post, we’ll describe how Pantacor Fleet can implement modern deployment strategies like canary and A/B testing. We’ll also showcase a new and powerful feature that enables engineers to control and customize their product’s firmware with containers without re-building the manufacturer’s BSP. 

What is Pantacor Fleet?

Fleet is a significant part of our SaaS, Pantacor Hub, enabling you to select, logically group, and update devices en masse. For example, using meta-data and rules, you can select and target specific versions of a particular device and group them into channels before applying your updates. Also, with advanced rule-based scheduling and other git-like functions such as PVR device state revision cloning and merging, you have modern automated embedded-first tooling that works with your CICD pipelines for device management.

Benefits of automated device fleet management

Reducing device management complexity through automation keeps IoT fleets up to date and secure. Embedded-first automation also provides the safety guards with the ability to roll back or roll forward, enabling developers to confidently release new services and features to their customers.

These are some of the other benefits of automated device fleet management:

  • Greater security guarantees – Act quickly and administer device CVEs and other vulnerabilities with patches and application updates.
  • Meet compliance regulations – Regulatory bodies are now mandating that devices are easily updated.Modular, easily updated software and firmware that can implement automated pipelines are vital components that enable teams to meet this requirement.
  • Increased productivity over product lifecycles – Strategically deliver new features continuously using modern automated workflows and advanced deployment strategies like canaries A/B testing.
  • Improved reliability and stability – With Pantavisor’s git-like state revision control, rolling back and forward to a good state is always possible; devices are never bricked and can also revert to a good state in the case of a bad deployment,

Deploying to device fleets with release channels

In order to deliver over-the-air deployments to device fleets with Pantacor, you will need to set up release channels. At a basic level, a channel is a rule-based group of devices. Once grouped into the channel, all devices are updated with the build or patch specified according to the schedule. As mentioned, fleets are grouped according to selection rules defined in the device meta-data. Once the devices are grouped by the rules in the channel, they are either automatically updated or updated by the specified schedule.

Implement advanced deployment strategies

Because Pantacor Fleet channel configuration is so flexible, it’s pretty powerful, and with a bit of imagination, you can also accomplish many different types of testing and deployment strategies. You could, for example, configure specific channels and their fleets to test your update before committing 100% to a live production environment. Or maybe you need to experiment with a few new features or fixes and restrict the rollout to a subset of devices before heading into production?

With Fleet it is simple to configure your channels to implement any of these advanced deployment strategies:

Canary deployments

A canary deployment allows you to release a new feature or a fix to a subset of users. Then, once you are satisfied with the deployment, you can gradually release to more of the fleet.

To implement this type of deployment, you could divide your large and extensive fleet into subsets by adding custom tags to the device meta-data and then by selecting on that data and creating a channel for each tag. For example, your fleet could be subdivided by the following meta-data tags: 10%, 25%, 75%, and 100%.

With the channel’s rule-based selection rules, each channel represents a portion of the canary deployment. Each of these channels can then be scheduled to run at specific time intervals, and then checked at each, until 100% of the fleet deploys.

A/B testing

Similar to and technically under the umbrella of canary are A/B deployments. In this strategy, you run two different experiments and then measure the results to see which one is best. Again with fleet channels, you can easily accomplish this by sub-dividing your fleet into two different groups and configuring each channel separately. You would then deploy them both simultaneously and measure the results.

Multi-repo firmware and application updates

After you’ve made updates and published them as a revision to Pantacor Hub, you can then proceed to release your updates through fleet channels. When creating a channel in Pantacor Fleet, you can add more than one repo URL(s) in the Source PVR URL field.

What does this mean and why use it?

This powerful feature enables product makers to add functionality to any base Board Support Package (BSP) with containers that run in the userland, and include apps from separately maintained repos. You could for example add an update URL from the vendor BSP repo, another for your own libraries and customizations and additional repos for your applications.


The containerized embedded system is assembled at release time before being deployed to the entire fleet from the release channel. When the release occurs (either automatically or on a schedule), containers are built from the specified repos and added to the userland. At the same time,  the underlying vendor BSP packages and firmware are also updated and deployed.

When you need to  update the components or the modules, they can be maintained in separate repos and even used to trigger a new release from each repo independently. By maintaining dual repos from source, you can customize any standard “vendor-issued” BSP with containers as well as have complete control over your extensions and add-ons without rebuilding the base BSP each time.

Final Thoughts

This post discussed how rules and release channels in Pantacor Fleet allow teams to experiment and test builds using advanced deployment strategies like canaries and A/B testing so that you can confidently add new features and deploy faster to millions of embedded Linux devices. In addition to this, we also introduced a feature that enables you to customize a standard vendor-issued BSP with containerized modules and also keep them up to date separately.

Pantavisor enables product makers to implement containers on embedded Linux systems. Take advantage of any standard Linux distro or specialty embedded Linux distros to define your product and then easily manage updates and patches across teams and product lines with containers. In addition, Pantavisor’s git-like container management tools lets you save and edit device states to share, deploy and secure fleets over the air from Pantacor Hub.

Try it for yourself with this easy-to-follow Getting Started Guide.