In this series of posts, we describe how to configure a simple and open source CI/CD pipeline for your embedded Linux projects. In this tutorial, we use GitLab and take advantage of Pantacor Hub and Pantavisor Linux.
| Table of Contents (4 part series) |
|---|
| Part 1: Automating CICD Pipelines for Embedded Linux Projects |
| Part 2 – Generate Flashable Images with CI/CD Pipelines |
| Part 3 – Templated CICD pipelines for Embedded Linux Devices |
| Part 4 – Customizing CICD pipelines for Embedded Linux Projects |
In this first part of a four part series on automating CICD pipelines for embedded Linux projects, we describe a basic setup that automatically keeps devices up to date with the latest and stable versions of your project’s code base.
Background
It happened again! Something stopped working and you have to probably spend long hours debugging. Was it the config file? Was it any of the latest fixes on the code landed by your team? An incompatibility with some dependency update? When did it stop working? Did it even work at some point? You could easily revert to the previous working security copy of your system. That could make it. After that, you just have to redo all the changes made by the team since the last security copy and everything will be fine. Wait… what were those changes?
This process could have been less painful if you had used an automated CI/CD pipeline (or Continuous Integration/ Continuous Deployment pipeline) in your device development workflow. One pipeline that would have added all the advantages of entirely working with distributed version control: traceability, change history in every file, automatic backup, branching and so on. It would have added the rest of continuous integration and development perks: shorter development life cycles, automatic building, testing and deployment, early detection of bugs, etc.
What you need
This tutorial assumes that you have gone through the prerequisites section in our documentation and installed the Pantavisor CLI pvr and signed up for an account at Pantacor Hub.
We will use the gpio management tutorial as a starting point. We are going to need the same material, though this is optional. In our case, we are going to double it so we can have a periodically updated device (latest version) and an on-demand updated device (stable version):
- 2 Raspberry Pi 3 B+
- 2 Power cable for RPi3
- 2 Internet-facing Ethernet cables
- Your computer
- Two or more LEDs
- over 50 ohm resistor for each LED
- One female to male breadboard jumper wires per LED plus an additional one for GND
- Breadboard
As already mentioned, it is not mandatory to set up the gpio hardware for this tutorial, but it is indeed a lot more fun 🙂
Using a virtual device
In case you don’t have the opportunity to get your hands on the hardware, or you simply don’t want to, you can substitute the claiming process with these commands:
pvr login
pvr dev create gpio-latest-device
pvr dev create gpio-stable-deviceThis will create a number of virtual devices in your Pantacor Hub account. The rest of the tutorial will be the same for you.
Lastly, if you are brave and you don’t have a Pi but you have another device, you can always try those. Our CI templates are agnostic to platform or architecture.
Claiming and committing to a device in Pantacor Hub
Our starting point will be a Raspberry Pi device flashed with our Pantavisor stable initial image. To see how to do the whole flashing and claiming process, go to our get started guide to see a step-by-step tutorial.
We are going to fork the code of our gpio app to take control of its source code. Wait for its pipeline to finish and install the resulting app with `pvr`. We will refer to this pipeline as the container-ci, to differentiate it from the device-ci, which is the one we are going to set up in this tutorial.
As soon as container-ci finishes, we will add it to our cloned device with pvr.
Then, we’ll commit and post the changes to Pantacor Hub. You can check that the device got updated in your device dashboard.
To complete the preliminary setup, check the history and list of installed apps in your Pantacor Hub device dashboard.
These are the apps that device-ci will keep up to date, getting the latest from that GitLab registry tag when the scheduled job is executed. They are just the default apps in the initial device you just flashed plus the gpio add you just installed.
Additionally, device-ci keeps the bsp (kernel, drivers and the Pantavisor binary) up to date with the latest stable version from our repositories.
Now, let’s double it!
Next, we will claim the other Raspberry Pi and post the contents of the previously set up device, so we can keep two (initially) identical devices: in the future, one will have the latest stuff from our development, the other one will keep the stable version and will be only updated when tagged.
This is the final physical set up, ready to roll.
Set up the GitLab project
Now, we are going to setup the proper device-ci. For this, we will need a new GitLab project.
There are a number of variables we need to set. For this tutorial we are going to only use PHUSER, PHPASS and PERSONAL_ACCESS_TOKEN. Check in the GitLab documentation how to create the last one (you will need api, readuser, readrepository and read_registry permissions). Don’t forget to mask all these variables!
The Pantacor Hub credentials are also needed so that the jobs can clone and post to your private Pantacor Hub devices.
The GitLab token is going to be used by the jobs to automatically push changes to the repository we just created when any of the installed apps or the bsp need an upgrade. This repo automatically creates a folder named recipes that contains a reference to the Pantacor Hub device and revision. In this way, tagging any of the states of the project will be possible to create stable versions on tested commits.
Import the template
Then, we clone the project and add the device-ci templates project as a submodule.
To configure the gitlab-ci.yml, we have added a tool called mkdevice-ci to aid you with the boring stuff. First, we need to init the repository to generate the configuration file.
You must manually edit that config file to add the information about your your devices. The device list expects a list of pairs of device names. It allows to keep a list of devices with different apps or even devices with different architectures.
In a real life setup, we may want this pair of devices to be different, so we can keep a running latest device and another running stable device per pair, but we are going to use the same device for simplicity sake. Just remember the first device in the pair will be updated on GitLab schedule, while the second one will be updated on manual tag pushing.
Next, use the install option of mkdevice-ci to generate the .gitlab-ci.yml file required by GitLab to run its CI. To complete the device-ci setup, just add, commit and push the changes to your repository.
Test it
In order to test the pipeline, we will push a change to the master branch in our previously forked gpio app project. For example, we are going to push a commit that makes the LED blinking instead of staying fixed when configured from Pantacor Hub metadata.
We can do that by substituting gpio.sh with the following code:
set_gpio_value() {
if [ ! -e $GPIO_PATH/gpio$1 ]; then
log "Exporting GPIO $1 with OUT direction"
echo $1 > $GPIO_PATH/export
echo "out" > $GPIO_PATH/gpio$1/direction
fi
if [ $2 = 1 ]; then
if [ `cat $GPIO_PATH/gpio$1/value` = 1 ]; then
echo 0 > $GPIO_PATH/gpio$1/value
log "Switching GPIO $1 value"
else
log "Switching GPIO $1 value"
echo 1 > $GPIO_PATH/gpio$1/value
fi
else
log "Setting GPIO $1 to 0"
echo 0 > $GPIO_PATH/gpio$1/value
fi
}Wait! we haven’t yet configured the scheduled job. You can do that with the help of GitLab UI. This scheduled job will be in charge of go through your list of configured devices and check for upgrades on the BSPs or apps. If any update is found, it will compile a list of changes and automatically post those changes to your devices.
The scheduler can be set as you wish. In this case we are going to update our latest devices daily. In part three of these series of tutorials, we are going to see how to override our templates and run the update job after a change in the app repository is pushed.
Manually trigger the scheduled job with the GitLab UI.
The pipeline is now running. The post job will check for updates in your device, while the push job will compile a list of changes and push changes to the device recipes, a list of files that contains the name and revision of the devices at each point.
The pipeline has found some changes! You can check those changes in the message of the commit that was just pushed.
After every automatic or manual commit push, a validation job will check if the device status is OK after the new revision is pushed to Pantacor Hub.
In the example, you can see stable at the left, with the old gpio app version. At the right we have the latest version with the blinking LEDs.
What’s next?
In the next part of these series, we are going to take a look at how to generate flashable images with the status of your device, so you can get factory images to install your project in your devices