2.6 Managing services and timers with systemd

Once the Linux kernel finishes initialising its device drivers and mounts the root file system (/), its final act in the boot sequence is to spawn the very first user-space process, Process ID 1 (PID 1), by executing /sbin/init.1
On almost every modern embedded Linux distribution (including Raspberry Pi OS, Debian, Ubuntu, and Yocto industrial builds), /sbin/init is a symbolic link to /usr/lib/systemd/systemd. As PID 1, systemd is the ancestor of every other process on the gateway: it mounts the remaining file systems in /etc/fstab, brings up network interfaces, starts background daemons (sshd, mosquitto, avahi-daemon), captures standard output and error streams into a structured binary journal, and restarts your gateway application automatically if it ever crashes.2
For an embedded systems engineer, mastering systemd is essential: a connected gateway in the field has no human sitting at an SSH terminal to start a C++ program after a power cut.
From sequential SysVinit to parallel systemd
Section titled “From sequential SysVinit to parallel systemd”To appreciate why systemd works the way it does, it helps to contrast it briefly with the classic UNIX SysVinit system used in older embedded distributions (and still found in minimal BusyBox builds):
- Classic
SysVinit: Used numbered runlevels (0= halt,1= single-user,2–5= multi-user/graphical,6= reboot) and executed shell scripts in/etc/init.d/strictly one after another in serial order (S01...,S02...). On a quad-core Arm processor, starting 40 shell scripts one after another leaves three of the four cores mostly idle during boot, andSysVinitdoes not supervise a daemon once its script has returned, so a crash ten minutes later goes unnoticed. - Modern
systemd: Replaces numbered runlevels with declarative Target units (.target) and replaces the/etc/init.d/shell scripts with declarative Service units (.service). Using socket and D-Bus activation alongside an explicit dependency graph (After=,Wants=,Requires=),systemdstarts independent services in parallel and tracks every process that a service creates using Linux kernel control groups (cgroups).2
Targets and freeing RAM on a headless gateway
Section titled “Targets and freeing RAM on a headless gateway”Table 2.6 shows how systemd targets correspond to classic runlevels and how you switch between them.
Table 2.6. Key systemd target units on an embedded Linux gateway.3
systemd Target Unit | Classic Runlevel | Purpose on an Embedded Gateway |
|---|---|---|
poweroff.target | 0 | Cleanly stops all services, unmounts file systems, and halts the system. |
rescue.target | 1 (S) | Single-user root maintenance shell for repairing broken systems. |
multi-user.target | 2, 3, 4 | Standard headless gateway state: full networking and services active, no GUI desktop compositor running. |
graphical.target | 5 | Starts multi-user.target plus the Wayland (labwc) desktop environment. |
reboot.target | 6 | Cleanly stops services, syncs disks, and resets the SoC. |
If you installed the full Raspberry Pi OS Desktop image on your gateway, it boots into graphical.target by default, spending 300–450 MB of RAM on the desktop environment even when no HDMI monitor is plugged in.
You can switch a running board immediately to headless mode, and make headless mode the permanent boot default, with two commands:4
derek@rpi:~ $ systemctl get-defaultgraphical.target# Immediately stop the desktop GUI without rebooting:derek@rpi:~ $ sudo systemctl isolate multi-user.target# Make headless multi-user mode the permanent default on future boots:derek@rpi:~ $ sudo systemctl set-default multi-user.targetCreated symlink '/etc/systemd/system/default.target' → '/usr/lib/systemd/system/multi-user.target'.Profiling boot time with systemd-analyze
Section titled “Profiling boot time with systemd-analyze”When designing an embedded appliance, boot time matters. systemd includes a built-in profiler, systemd-analyze, that breaks down exactly how many milliseconds were spent in the kernel versus user space, and which service held up the boot chain:6
derek@rpi:~ $ systemd-analyze timeStartup finished in 1.942s (kernel) + 6.318s (userspace) = 8.260smulti-user.target reached after 6.190s in userspace.
# List the slowest services during boot:derek@rpi:~ $ systemd-analyze blame | head -n 52.412s NetworkManager-wait-online.service1.104s raspi-config.service 682ms systemd-udev-settle.service 415ms user@1000.service 310ms avahi-daemon.serviceEveryday service management (systemctl and journalctl)
Section titled “Everyday service management (systemctl and journalctl)”Table 2.7 summarises the systemctl and journalctl commands you will use daily when operating a gateway or debugging your own C++ daemons.
Table 2.7. Essential systemctl and journalctl commands for managing gateway services.47
| Command | Action |
|---|---|
systemctl status <unit> | Show current state (active (running), failed), PID, memory usage, cgroup tree, and the last 10 log lines. |
sudo systemctl start <unit> | Start a service immediately in the current session (does not persist across reboots). |
sudo systemctl stop <unit> | Send SIGTERM to cleanly stop a running service right now. |
sudo systemctl restart <unit> | Stop and then start a service (use after updating a binary; use try-restart to restart only if already running). |
sudo systemctl reload <unit> | Ask a running service to reload its configuration without stopping. This runs the unit’s ExecReload= command (often a SIGHUP)8 and fails if the unit defines none. |
sudo systemctl enable --now <unit> | Create boot symlinks so the service starts automatically at boot and start it immediately right now. |
sudo systemctl disable --now <unit> | Remove boot symlinks and stop the service immediately. |
sudo systemctl daemon-reload | Mandatory after editing any .service or .timer file so systemd re-parses the unit definitions on disk. |
journalctl -u <unit> -f | Follow (-f, like tail -f) live log output from a specific service in real time. |
journalctl -u <unit> -b -p err | Show only error-priority (-p err) and higher messages from <unit> since the current boot (-b). |
Writing a custom .service unit for your own C++ program
Section titled “Writing a custom .service unit for your own C++ program”Suppose you have compiled a C++ gateway application (/usr/local/bin/sensor-logger) that reads sensor telemetry and publishes it over the network. How do you make it start automatically on boot, wait until the network is actually connected, run under an unprivileged user account instead of root, and restart automatically if an unhandled exception or network glitch causes it to exit?
Never edit files inside /usr/lib/systemd/system/ (those belong to apt packages and will be overwritten on upgrades).9 Instead, create your own unit files in /etc/systemd/system/. Here is a template (/etc/systemd/system/sensor-logger.service):
[Unit]Description=EEN1071 Gateway Sensor Telemetry LoggerDocumentation=https://book.derekmolloy.ie/# Wait until the network interface is configured and has an IP address:After=network-online.targetWants=network-online.target
[Service]Type=simple# Run as an unprivileged service account, NOT root (Section 2.7):User=sensorsvcGroup=sensorsvc# Create /var/lib/sensor-logger, owned by sensorsvc and writable# under ProtectSystem=strict:StateDirectory=sensor-loggerWorkingDirectory=/var/lib/sensor-loggerExecStart=/usr/local/bin/sensor-logger --interval 5000
# Automatically restart if the process crashes or exits with an error:Restart=on-failureRestartSec=5s
# Optional software watchdog (Chapter 20). The program must then call# sd_notify(0, "WATCHDOG=1") more often than this, or systemd restarts it:# WatchdogSec=30s
# Standard output and error go automatically to systemd-journald:StandardOutput=journalStandardError=journalSyslogIdentifier=sensor-logger
# Basic security hardening (sandboxing):NoNewPrivileges=trueProtectSystem=strictPrivateTmp=true
[Install]WantedBy=multi-user.targetOnce saved, register and start the service in three lines:
derek@rpi:~ $ sudo systemctl daemon-reloadderek@rpi:~ $ sudo systemctl enable --now sensor-logger.servicederek@rpi:~ $ systemctl status sensor-logger.serviceNotice what happens when your C++ program writes to std::cout or std::cerr: you do not need to manage log file descriptors or risk filling up the SD card. systemd captures every line written to stdout/stderr, attaches a timestamp and the process ID, and stores it in journald,2 where you can inspect it with journalctl -u sensor-logger -f.
One caveat: when stdout is not a terminal, the C and C++ libraries buffer it in blocks,10 so lines written to std::cout can reach the journal late, or not at all if the program crashes. End each log line with std::endl (or call std::flush), or write status messages to std::cerr, which is not buffered.
Figure 2.13 runs this unit through a boot. Remove the network ordering, change Restart= and RestartSec=, and make the program crash or exit, then compare the timeline, the journal and the final systemctl status.
Boot order and supervision of sensor-logger.service
Change the unit file and what the program does; the timeline and journal follow.
[Unit] After=network-online.target Wants=network-online.target [Service] ExecStart=/usr/local/bin/sensor-logger --interval 5000 Restart=on-failure RestartSec=5s
[ 2.100000] rpi kernel: Run /sbin/init as init process [ 3.400000] rpi systemd[1]: Reached target basic.target - Basic System. [ 3.600000] rpi systemd[1]: Reached target network.target - Network. [ 8.200000] rpi systemd[1]: Reached target network-online.target - Network is Online. [ 8.200000] rpi systemd[1]: Started sensor-logger.service - EEN1071 Gateway Sensor Telemetry Logger. [ 8.250000] rpi sensor-logger[612]: Connected to broker.local:1883, publishing every 5 s [ 8.300000] rpi systemd[1]: Reached target multi-user.target - Multi-User System.
Boot times are illustrative. The start limit is systemd's default: 5 starts in 10 s.
Figure 2.13. sensor-logger.service during boot and supervision. Without After=network-online.target the daemon starts before the network is up; with the default RestartSec=100ms8 its restarts reach the start limit (5 starts in 10 s)11 before the network arrives, and systemd stops trying.
Periodic tasks: systemd timers vs classic cron
Section titled “Periodic tasks: systemd timers vs classic cron”Often on a gateway, rather than running a continuous while (true) daemon loop, you want to run a short task periodically: for example, sampling a slow environmental sensor every 5 minutes, rotating a database backup at 03:00 every night, or pushing a health report every 15 minutes. Linux gives you two ways to schedule periodic jobs:
1. The modern way: systemd .timer units
Section titled “1. The modern way: systemd .timer units”A systemd timer pairs a <name>.timer file with a <name>.service file (where the .service has Type=oneshot and no [Install] section because the timer triggers it).12 For example, /etc/systemd/system/health-check.timer:
[Unit]Description=Run Gateway Health Check Every 5 Minutes
[Timer]# First run 1 minute after boot, then every 5 minutes thereafter:OnBootSec=1minOnUnitActiveSec=5min# Or use calendar syntax (like cron): OnCalendar=*-*-* *:00/15:00RandomizedDelaySec=10sUnit=health-check.service
[Install]WantedBy=timers.targetEnable the timer with sudo systemctl enable --now health-check.timer and inspect all upcoming scheduled runs across the gateway with systemctl list-timers. Because each run executes as a real systemd service, its exit status, CPU time, and journalctl logs are tracked automatically.
2. The familiar UNIX classic: cron and crontab
Section titled “2. The familiar UNIX classic: cron and crontab”You will still encounter cron on most UNIX and embedded Linux systems.
The cron daemon wakes up once every minute, checks system crontabs (/etc/crontab, /etc/cron.d/, /etc/cron.hourly/) and per-user crontabs (edited with crontab -e and listed with crontab -l), and runs any command whose five time fields match the current clock:13
# User crontab syntax (edit with: crontab -e)# ┌───────────── minute (0 - 59)# │ ┌───────────── hour (0 - 23)# │ │ ┌───────────── day of the month (1 - 31)# │ │ │ ┌───────────── month (1 - 12)# │ │ │ │ ┌───────────── day of the week (0 - 6, Sunday=0)# │ │ │ │ │*/15 * * * * /usr/local/bin/publish_telemetry >/dev/null 2>&1@reboot /usr/local/bin/on_boot_init.shControlling onboard LEDs via sysfs and adding a soft power button
Section titled “Controlling onboard LEDs via sysfs and adding a soft power button”The /sys/class/leds/ kernel interface
Section titled “The /sys/class/leds/ kernel interface”In Lab 1, you briefly blinked the onboard green activity LED (ACT). Let us look closer at how the Linux kernel’s leds class driver exposes onboard LEDs through the sysfs virtual file system at /sys/class/leds/:14
derek@rpi:~ $ ls /sys/class/leds/ACT PWR mmc0::derek@rpi:~ $ ls -l /sys/class/leds/ACT/total 0-rw-r--r-- 1 root root 4096 Nov 7 14:10 brightness-r--r--r-- 1 root root 4096 Nov 7 14:10 max_brightness-rw-r--r-- 1 root root 4096 Nov 7 14:10 trigger(Note: Unlike the old /sys/class/gpio/ interface (deprecated and replaced by /dev/gpiochip* and libgpiod v2 in Chapter 6), the /sys/class/leds/ subsystem is a supported kernel class interface designed for status, activity, and backlight LEDs).
Reading the trigger file shows every kernel event source that can drive the LED automatically, with the currently active trigger enclosed in square brackets ([mmc0] = flash on microSD card activity):
derek@rpi:~ $ cat /sys/class/leds/ACT/triggernone rc-feedback kbd-scrolllock kbd-numlock kbd-capslock kbd-kanalock kbd-shiftlock kbd-altgrlock kbd-ctrllock kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock kbd-ctrlrlock timer oneshot heartbeat backlight gpio cpu cpu0 cpu1 cpu2 cpu3 default-on input panic actpwr [mmc0] rfkill-any rfkill-noneWhen you write timer into trigger, the kernel LED driver creates two new files in that directory, delay_on and delay_off (in milliseconds).14 The kernel then blinks the LED itself, with no user-space process running:
derek@rpi:~ $ echo timer | sudo tee /sys/class/leds/ACT/triggertimerderek@rpi:~ $ ls /sys/class/leds/ACT/brightness delay_off delay_on max_brightness power subsystem trigger uevent# Blink fast (100 ms ON, 100 ms OFF = 5 Hz):derek@rpi:~ $ echo 100 | sudo tee /sys/class/leds/ACT/delay_onderek@rpi:~ $ echo 100 | sudo tee /sys/class/leds/ACT/delay_off# Or set the heartbeat trigger (pulses like a clinical ECG; speeds up as CPU load increases!):derek@rpi:~ $ echo heartbeat | sudo tee /sys/class/leds/ACT/trigger# Restore normal microSD activity behaviour:derek@rpi:~ $ echo mmc0 | sudo tee /sys/class/leds/ACT/triggerFigure 2.14 lets you try these commands without a board, and shows which process opens the file in each form of the command.
The ACT LED through /sys/class/leds/, and who opens the file
Run the commands in order and watch the LED, the file list and the terminal.
mmc0- -rw-r--r-- root rootbrightness= set by the trigger
- -r--r--r-- root rootmax_brightness
- -rw-r--r-- root roottrigger= [mmc0]
After boot the trigger is mmc0. Start by reading the trigger file.
derek@rpi:~ $
Not modelled: the PWR LED and differences between boards, the full trigger list, the oneshot and cpu triggers, brightness values above 1, and the heartbeat speeding up with CPU load.
Figure 2.14. The green ACT LED of the Raspberry Pi 4 driven through /sys/class/leds/ACT/. The kernel’s timer, heartbeat and mmc0 triggers blink it with no user-space process running, and a write to a root-owned sysfs file succeeds only when the process that opens the file is allowed to write it.
Try this:
- Click
cat .../trigger, then thetimertrigger button:delay_onanddelay_off(500 mseach) appear in the file list. Click100 / 100and read5 Hzat a 50% duty cycle, then50 / 950for1 Hzat 5%. - Click
heartbeat: the delay files go, and the trace shows two short flashes in each period. Click1 (on)and nothing lasting happens, because the trigger still controls the LED. Clicknone, then1 (on)and0 (off), and the LED follows. - Switch to the second view, on why
sudo echo >fails. Step throughsudo echo >,echo | sudo teeandsudo sh -c '...'and note which process is marked as the one that opens the file. Then chooseecho >with nosudoand tick theudevrule: the write works with nosudo(Section 2.7).
The figure does not model the PWR LED, the full trigger list, brightness values above 1, or the heartbeat speeding up with CPU load, and its mmc0 flicker is a fixed pattern.
A zero-code hardware shutdown button (gpio-shutdown)
Section titled “A zero-code hardware shutdown button (gpio-shutdown)”In Section 2.2, we noted that you must never power off a Linux gateway by pulling the USB-C cable while the root file system is mounted read-write. The Raspberry Pi 5 has a power button on the PCB beside the status LED. A short press starts a clean systemd shutdown on Raspberry Pi OS Lite (the desktop asks for a second press to confirm).15
What about a headless Raspberry Pi 4 mounted inside an enclosure with no keyboard or SSH screen nearby? Older tutorials use a background shell or Python script that polls a GPIO pin and calls /sbin/shutdown -h now. That is no longer needed. Add one Devicetree overlay line to /boot/firmware/config.txt:
dtoverlay=gpio-shutdown,gpio_pin=17The overlay’s defaults (active_low=1, gpio_pull=up) suit a button to ground.16 Wire a momentary push-button between Pin 11 (GPIO17) and Pin 9 (GND) and reboot. When a user presses the button:
- The kernel’s
gpio-keysinput driver detects the falling edge onGPIO17and generates a standard LinuxKEY_POWERevent.16 systemd-logindreceivesKEY_POWERand initiates a clean, orderlysystemctl poweroff, stopping all services and unmounting the microSD card before halting the SoC. WhyGPIO17and not the overlay’s default pin,GPIO3(Pin 5)? Many tutorials useGPIO3because, withWAKE_ON_GPIO=1in the bootloader configuration (the default, shown in Section 2.5), pullingGPIO3low while a Pi 4 is halted starts the board again, so one button can both shut it down and wake it.15 However,GPIO3isSCL1, the clock line of the I²C bus that we use for sensors in later chapters, and a pin cannot be both. With the button onGPIO17, restart a halted board by cycling its power.
🧩 Knowledge Check
Section titled “🧩 Knowledge Check”Match the unit file line to what it does
Why does sudo echo none > /sys/class/leds/ACT/trigger fail with Permission denied?
You change RestartSec= in /etc/systemd/system/sensor-logger.service. What must you run before the next restart uses the new value?
Which of the following are reasons to prefer a systemd timer to a cron job on a gateway?
Footnotes
Section titled “Footnotes”-
D. Molloy, Exploring Raspberry Pi: Interfacing to the Real World with Embedded Linux, Wiley, 2016, ch. 3, “Exploring Embedded Linux Systems.” ISBN 978-1-119-18868-1. ↩
-
systemd project, systemd(1): systemd system and service manager, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.html. ↩ ↩2 ↩3
-
systemd project, runlevel(8), Table 1, “Mapping between runlevels and systemd targets”. Available: man7.org/linux/man-pages/man8/runlevel.8.html. ↩
-
systemd project, systemctl(1): control the systemd system and service manager, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemctl.html. ↩ ↩2
-
US National Highway Traffic Safety Administration, Federal Motor Vehicle Safety Standard No. 111: Rear visibility, 49 CFR 571.111, S5.5.3 (response time), required for all new light vehicles from 1 May 2018. Available: ecfr.gov. ↩
-
systemd project, systemd-analyze(1): analyze and debug system manager, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd-analyze.html. ↩
-
systemd project, journalctl(1): print log entries from the systemd journal, systemd 262. Available: freedesktop.org/software/systemd/man/latest/journalctl.html. ↩
-
systemd project, systemd.service(5): service unit configuration, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.service.html. ↩ ↩2
-
systemd project, systemd.unit(5): unit configuration, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.unit.html. ↩
-
Linux man-pages project, setbuf(3): stream buffering operations, man-pages 6.19. Available: man7.org/linux/man-pages/man3/setbuf.3.html. ↩
-
systemd project, systemd-system.conf(5),
DefaultStartLimitIntervalSec=andDefaultStartLimitBurst=, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd-system.conf.html. ↩ -
systemd project, systemd.timer(5): timer unit configuration, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.timer.html. ↩
-
Debian, cron(8): daemon to execute scheduled commands, cron 3.0pl1, Debian trixie. Available: manpages.debian.org/trixie/cron/cron.8.en.html. ↩ ↩2
-
The Linux Kernel documentation, LED handling under Linux. Available: docs.kernel.org/leds/leds-class.html. ↩ ↩2
-
Raspberry Pi Ltd, Raspberry Pi hardware, sections “Power button” and “Raspberry Pi bootloader configuration” (
WAKE_ON_GPIO). Available: raspberrypi.com/documentation/computers/raspberry-pi.html. ↩ ↩2 -
Raspberry Pi Ltd, Device Tree overlays README, entry
gpio-shutdown, raspberrypi/firmware repository. Available: github.com/raspberrypi/firmware/blob/master/boot/overlays/README. ↩ ↩2
© 2026 Derek Molloy, Dublin City University. All rights reserved.