Few of the millions of embedded systems connected and running in people’s homes and businesses can deliver meaningful logs. Yet, logs are essential for maintaining, monitoring, and troubleshooting device fleets so operators can drill down and promptly solve issues.
This post discusses how the Pantavisor logging system works in embedded Linux devices. We’ll show you how device logs routed and aggregated with Pantacor Hub can be analyzed in Elastic. You’re not limited to Elastic; with Pantacor Hub APIs, you can also choose to use Fluentd or any other favorite logging aggregators or observability dashboards like Grafana. On the telemetry side, a Pantavisor-enabled fleet can take advantage of InfluxDB with the Telegraf plugin for real-time IoT fleet monitoring. In this post, we’ll provide an overview of the available logs and discuss the various ways to drill down on them so that you can quickly troubleshoot and fix a problem.
Logs available in embedded systems built with Pantavisor
One of the most powerful features offered by Pantavisor – besides its ability to provide a fully containerized environment on embedded Linux devices, including those that are low spec – is its comprehensive logging environment. Without any configuration, you can access the following types of logs:
- Stdout logs from containers – Standard container and application logs and messages.
- LXC (app startup) logs and console logs – Drill down on and analyze logs from the device, Pantavisor, and the Linux system.
In addition to its logging coverage, Pantavisor also secures all logs and maintains them with each state revision.
How Pantavisor logging works
Logs run through the Pantacor Hub API endpoints (at pantahub /logs ) but also provide options to configure the log level and location where the logs get stored. Pantavisor can centralize both its own logs and your container logs, separated by revision, in one place on the disk.
Containers on Pantavisor also direct their logs through a small server that offers a socket (pv-ctrl-log). By default, containers direct the following logs to the log server:
- syslog
- messages
- lxc logs
- lxc console
Additional files may also be added to the JSON state file with configurable log parameters such as log.capture, log.maxsize, log.level.
Pushing additional logs to Pantacor Hub
You can specify any additional log files and push them to Pantacor Hub by adding the “logs” object to the JSON state file. In the example below, alpine and NGINX are configured to push their logs to Pantacor Hub:
{
"#spec": "service-manifest-run@1",
"config": "lxc.container.conf",
"name":"alpine-hotspot",
"storage":{
"lxc-overlay" : {
"persistence": "boot"
}
},
"type":"lxc",
"root-volume": "root.squashfs",
"volumes":[],
"logs": [
{"file":"/var/log/syslog","maxsize":102485760,"truncate":true,"name":"alpine-logger"},
{"file":"/www/log/wwwlog","maxsize":102485760,"truncate":true,"name":"nginx-logger"},
{"lxc":"enable","maxsize":409943040,"truncate":true,"name":"alpine-lxc"}
]
}
See, Pushing App Logs to Pantacor Hub for more information.
Viewing logs
As mentioned, there are multiple ways in which to view logs from Pantavisor-enabled fleets:
- From the console within the Pantacor Hub SaaS.
- Tailed and interacted with at the device level from a terminal window.
- From popular log management and dashboard tools such as Elastic Search, InFluxDB, Grafana, and others.
Pantacor Hub Dashboard
Once you’ve installed Pantavisor and claimed the device, basic logs from any of the containers running on the device can be monitored, filtered, and observed directly from within the Pantacor Hub dashboard:
Terminal window
Logs can also be tailed after ssh’ing onto a device and pulling the logs from Pantacor Hub with:
pvr device logs
Filter on the log fields from the command line with several different PVR CLI options or with a Golang template and Sprig: functions:
pvr device logs [--template=<short|json|] [/source][@level][#platform]
Viewing logs in Elastic
When viewing from Elastic, the Pantacor Hub APIs are responsible for routing and aggregating all device logs. You can then use the Elastic dashboard to analyze, drill down, troubleshoot a problem, or even monitor your device fleet.
Below, I’ve filtered messages from the running ADU agent from Azure IoT that is running on our fleet:
Telegraf plugin logs
In this example, we filter on the Telegraf plugin in the Pantacor Hub API. Telegraf aggregates telemetry and connects to InfluxDB for in-depth analysis of metrics collected from your device fleets. You can then use the InfluxDB time-series database to drill down on telemetry and also set up alerts to monitor entire fleets in real-time.
(Stay tuned for a tutorial on implementing Telegraf and InfluxDB on Pantavisor-enabled devices.
Final Thoughts
In this post, we discussed how logging in Pantavisor works and how you can view and manage logs.
The Pantacor platform enables developers and operators to manage software lifecycle updates using modern open source tools and DevOps workflows on any embedded Linux device. Data returned from devices such as logs and telemetry are centralized and routed through Pantacor Hub API. In addition, Pantacor implements open standards, and data can be more easily analyzed and acted on using your choice of tools available from the cloud-native ecosystem.
Get started with Pantacor and Pantavisor with this tutorial or contact us and we’d be happy to walk you through the platform.