Skip to content

2.6 Managing services and timers with systemd

Once the Linux kernel finishes initialising its device drivers and mounts the root file system (/), its final act in the boot sequence is to spawn the very first user-space process, Process ID 1 (PID 1), by executing /sbin/init.1

On almost every modern embedded Linux distribution (including Raspberry Pi OS, Debian, Ubuntu, and Yocto industrial builds), /sbin/init is a symbolic link to /usr/lib/systemd/systemd. As PID 1, systemd is the ancestor of every other process on the gateway: it mounts the remaining file systems in /etc/fstab, brings up network interfaces, starts background daemons (sshd, mosquitto, avahi-daemon), captures standard output and error streams into a structured binary journal, and restarts your gateway application automatically if it ever crashes.2

For an embedded systems engineer, mastering systemd is essential: a connected gateway in the field has no human sitting at an SSH terminal to start a C++ program after a power cut.


From sequential SysVinit to parallel systemd

Section titled “From sequential SysVinit to parallel systemd”

To appreciate why systemd works the way it does, it helps to contrast it briefly with the classic UNIX SysVinit system used in older embedded distributions (and still found in minimal BusyBox builds):

  • Classic SysVinit: Used numbered runlevels (0 = halt, 1 = single-user, 2–5 = multi-user/graphical, 6 = reboot) and executed shell scripts in /etc/init.d/ strictly one after another in serial order (S01..., S02...). On a quad-core Arm processor, starting 40 shell scripts one after another leaves three of the four cores mostly idle during boot, and SysVinit does not supervise a daemon once its script has returned, so a crash ten minutes later goes unnoticed.
  • Modern systemd: Replaces numbered runlevels with declarative Target units (.target) and replaces the /etc/init.d/ shell scripts with declarative Service units (.service). Using socket and D-Bus activation alongside an explicit dependency graph (After=, Wants=, Requires=), systemd starts independent services in parallel and tracks every process that a service creates using Linux kernel control groups (cgroups).2

Targets and freeing RAM on a headless gateway

Section titled “Targets and freeing RAM on a headless gateway”

Table 2.6 shows how systemd targets correspond to classic runlevels and how you switch between them.

Table 2.6. Key systemd target units on an embedded Linux gateway.3

systemd Target UnitClassic RunlevelPurpose on an Embedded Gateway
poweroff.target0Cleanly stops all services, unmounts file systems, and halts the system.
rescue.target1 (S)Single-user root maintenance shell for repairing broken systems.
multi-user.target2, 3, 4Standard headless gateway state: full networking and services active, no GUI desktop compositor running.
graphical.target5Starts multi-user.target plus the Wayland (labwc) desktop environment.
reboot.target6Cleanly stops services, syncs disks, and resets the SoC.

If you installed the full Raspberry Pi OS Desktop image on your gateway, it boots into graphical.target by default, spending 300–450 MB of RAM on the desktop environment even when no HDMI monitor is plugged in.

You can switch a running board immediately to headless mode, and make headless mode the permanent boot default, with two commands:4

Terminal window
derek@rpi:~ $ systemctl get-default
graphical.target
# Immediately stop the desktop GUI without rebooting:
derek@rpi:~ $ sudo systemctl isolate multi-user.target
# Make headless multi-user mode the permanent default on future boots:
derek@rpi:~ $ sudo systemctl set-default multi-user.target
Created symlink '/etc/systemd/system/default.target' → '/usr/lib/systemd/system/multi-user.target'.

When designing an embedded appliance, boot time matters. systemd includes a built-in profiler, systemd-analyze, that breaks down exactly how many milliseconds were spent in the kernel versus user space, and which service held up the boot chain:6

Terminal window
derek@rpi:~ $ systemd-analyze time
Startup finished in 1.942s (kernel) + 6.318s (userspace) = 8.260s
multi-user.target reached after 6.190s in userspace.
# List the slowest services during boot:
derek@rpi:~ $ systemd-analyze blame | head -n 5
2.412s NetworkManager-wait-online.service
1.104s raspi-config.service
682ms systemd-udev-settle.service
415ms user@1000.service
310ms avahi-daemon.service

Everyday service management (systemctl and journalctl)

Section titled “Everyday service management (systemctl and journalctl)”

Table 2.7 summarises the systemctl and journalctl commands you will use daily when operating a gateway or debugging your own C++ daemons.

Table 2.7. Essential systemctl and journalctl commands for managing gateway services.47

CommandAction
systemctl status <unit>Show current state (active (running), failed), PID, memory usage, cgroup tree, and the last 10 log lines.
sudo systemctl start <unit>Start a service immediately in the current session (does not persist across reboots).
sudo systemctl stop <unit>Send SIGTERM to cleanly stop a running service right now.
sudo systemctl restart <unit>Stop and then start a service (use after updating a binary; use try-restart to restart only if already running).
sudo systemctl reload <unit>Ask a running service to reload its configuration without stopping. This runs the unit’s ExecReload= command (often a SIGHUP)8 and fails if the unit defines none.
sudo systemctl enable --now <unit>Create boot symlinks so the service starts automatically at boot and start it immediately right now.
sudo systemctl disable --now <unit>Remove boot symlinks and stop the service immediately.
sudo systemctl daemon-reloadMandatory after editing any .service or .timer file so systemd re-parses the unit definitions on disk.
journalctl -u <unit> -fFollow (-f, like tail -f) live log output from a specific service in real time.
journalctl -u <unit> -b -p errShow only error-priority (-p err) and higher messages from <unit> since the current boot (-b).

Writing a custom .service unit for your own C++ program

Section titled “Writing a custom .service unit for your own C++ program”

Suppose you have compiled a C++ gateway application (/usr/local/bin/sensor-logger) that reads sensor telemetry and publishes it over the network. How do you make it start automatically on boot, wait until the network is actually connected, run under an unprivileged user account instead of root, and restart automatically if an unhandled exception or network glitch causes it to exit?

Never edit files inside /usr/lib/systemd/system/ (those belong to apt packages and will be overwritten on upgrades).9 Instead, create your own unit files in /etc/systemd/system/. Here is a template (/etc/systemd/system/sensor-logger.service):

/etc/systemd/system/sensor-logger.service
[Unit]
Description=EEN1071 Gateway Sensor Telemetry Logger
Documentation=https://book.derekmolloy.ie/
# Wait until the network interface is configured and has an IP address:
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
# Run as an unprivileged service account, NOT root (Section 2.7):
User=sensorsvc
Group=sensorsvc
# Create /var/lib/sensor-logger, owned by sensorsvc and writable
# under ProtectSystem=strict:
StateDirectory=sensor-logger
WorkingDirectory=/var/lib/sensor-logger
ExecStart=/usr/local/bin/sensor-logger --interval 5000
# Automatically restart if the process crashes or exits with an error:
Restart=on-failure
RestartSec=5s
# Optional software watchdog (Chapter 20). The program must then call
# sd_notify(0, "WATCHDOG=1") more often than this, or systemd restarts it:
# WatchdogSec=30s
# Standard output and error go automatically to systemd-journald:
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sensor-logger
# Basic security hardening (sandboxing):
NoNewPrivileges=true
ProtectSystem=strict
PrivateTmp=true
[Install]
WantedBy=multi-user.target

Once saved, register and start the service in three lines:

Terminal window
derek@rpi:~ $ sudo systemctl daemon-reload
derek@rpi:~ $ sudo systemctl enable --now sensor-logger.service
derek@rpi:~ $ systemctl status sensor-logger.service

Notice what happens when your C++ program writes to std::cout or std::cerr: you do not need to manage log file descriptors or risk filling up the SD card. systemd captures every line written to stdout/stderr, attaches a timestamp and the process ID, and stores it in journald,2 where you can inspect it with journalctl -u sensor-logger -f.

One caveat: when stdout is not a terminal, the C and C++ libraries buffer it in blocks,10 so lines written to std::cout can reach the journal late, or not at all if the program crashes. End each log line with std::endl (or call std::flush), or write status messages to std::cerr, which is not buffered.

Figure 2.13 runs this unit through a boot. Remove the network ordering, change Restart= and RestartSec=, and make the program crash or exit, then compare the timeline, the journal and the final systemctl status.

Figure 2.13

Boot order and supervision of sensor-logger.service

Change the unit file and what the program does; the timeline and journal follow.

Restart=
RestartSec=
The program
[Unit]
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/local/bin/sensor-logger --interval 5000
Restart=on-failure
RestartSec=5s
0 s2 s4 s6 s8 s10 s12 s14 skernel / targetsPID 1basiconlineNetworkManagernetwork onlinesensor-logger
running and publishing running, no data sent✕ process exited with an error
● sensor-logger.service
Active: active (running)
Running since it was first started. Publishing for 112 s of the first 120 s.
journalctl -b -o short-monotonic (excerpt)
[   2.100000] rpi kernel: Run /sbin/init as init process
[   3.400000] rpi systemd[1]: Reached target basic.target - Basic System.
[   3.600000] rpi systemd[1]: Reached target network.target - Network.
[   8.200000] rpi systemd[1]: Reached target network-online.target - Network is Online.
[   8.200000] rpi systemd[1]: Started sensor-logger.service - EEN1071 Gateway Sensor Telemetry Logger.
[   8.250000] rpi sensor-logger[612]: Connected to broker.local:1883, publishing every 5 s
[   8.300000] rpi systemd[1]: Reached target multi-user.target - Multi-User System.

Boot times are illustrative. The start limit is systemd's default: 5 starts in 10 s.

Figure 2.13. sensor-logger.service during boot and supervision. Without After=network-online.target the daemon starts before the network is up; with the default RestartSec=100ms8 its restarts reach the start limit (5 starts in 10 s)11 before the network arrives, and systemd stops trying.


Periodic tasks: systemd timers vs classic cron

Section titled “Periodic tasks: systemd timers vs classic cron”

Often on a gateway, rather than running a continuous while (true) daemon loop, you want to run a short task periodically: for example, sampling a slow environmental sensor every 5 minutes, rotating a database backup at 03:00 every night, or pushing a health report every 15 minutes. Linux gives you two ways to schedule periodic jobs:

A systemd timer pairs a <name>.timer file with a <name>.service file (where the .service has Type=oneshot and no [Install] section because the timer triggers it).12 For example, /etc/systemd/system/health-check.timer:

/etc/systemd/system/health-check.timer
[Unit]
Description=Run Gateway Health Check Every 5 Minutes
[Timer]
# First run 1 minute after boot, then every 5 minutes thereafter:
OnBootSec=1min
OnUnitActiveSec=5min
# Or use calendar syntax (like cron): OnCalendar=*-*-* *:00/15:00
RandomizedDelaySec=10s
Unit=health-check.service
[Install]
WantedBy=timers.target

Enable the timer with sudo systemctl enable --now health-check.timer and inspect all upcoming scheduled runs across the gateway with systemctl list-timers. Because each run executes as a real systemd service, its exit status, CPU time, and journalctl logs are tracked automatically.

2. The familiar UNIX classic: cron and crontab

Section titled “2. The familiar UNIX classic: cron and crontab”

You will still encounter cron on most UNIX and embedded Linux systems.

The cron daemon wakes up once every minute, checks system crontabs (/etc/crontab, /etc/cron.d/, /etc/cron.hourly/) and per-user crontabs (edited with crontab -e and listed with crontab -l), and runs any command whose five time fields match the current clock:13

# User crontab syntax (edit with: crontab -e)
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6, Sunday=0)
# │ │ │ │ │
*/15 * * * * /usr/local/bin/publish_telemetry >/dev/null 2>&1
@reboot /usr/local/bin/on_boot_init.sh

Controlling onboard LEDs via sysfs and adding a soft power button

Section titled “Controlling onboard LEDs via sysfs and adding a soft power button”

In Lab 1, you briefly blinked the onboard green activity LED (ACT). Let us look closer at how the Linux kernel’s leds class driver exposes onboard LEDs through the sysfs virtual file system at /sys/class/leds/:14

Terminal window
derek@rpi:~ $ ls /sys/class/leds/
ACT PWR mmc0::
derek@rpi:~ $ ls -l /sys/class/leds/ACT/
total 0
-rw-r--r-- 1 root root 4096 Nov 7 14:10 brightness
-r--r--r-- 1 root root 4096 Nov 7 14:10 max_brightness
-rw-r--r-- 1 root root 4096 Nov 7 14:10 trigger

(Note: Unlike the old /sys/class/gpio/ interface (deprecated and replaced by /dev/gpiochip* and libgpiod v2 in Chapter 6), the /sys/class/leds/ subsystem is a supported kernel class interface designed for status, activity, and backlight LEDs).

Reading the trigger file shows every kernel event source that can drive the LED automatically, with the currently active trigger enclosed in square brackets ([mmc0] = flash on microSD card activity):

Terminal window
derek@rpi:~ $ cat /sys/class/leds/ACT/trigger
none rc-feedback kbd-scrolllock kbd-numlock kbd-capslock kbd-kanalock kbd-shiftlock kbd-altgrlock kbd-ctrllock kbd-altlock kbd-shiftllock kbd-shiftrlock kbd-ctrlllock kbd-ctrlrlock timer oneshot heartbeat backlight gpio cpu cpu0 cpu1 cpu2 cpu3 default-on input panic actpwr [mmc0] rfkill-any rfkill-none

When you write timer into trigger, the kernel LED driver creates two new files in that directory, delay_on and delay_off (in milliseconds).14 The kernel then blinks the LED itself, with no user-space process running:

Terminal window
derek@rpi:~ $ echo timer | sudo tee /sys/class/leds/ACT/trigger
timer
derek@rpi:~ $ ls /sys/class/leds/ACT/
brightness delay_off delay_on max_brightness power subsystem trigger uevent
# Blink fast (100 ms ON, 100 ms OFF = 5 Hz):
derek@rpi:~ $ echo 100 | sudo tee /sys/class/leds/ACT/delay_on
derek@rpi:~ $ echo 100 | sudo tee /sys/class/leds/ACT/delay_off
# Or set the heartbeat trigger (pulses like a clinical ECG; speeds up as CPU load increases!):
derek@rpi:~ $ echo heartbeat | sudo tee /sys/class/leds/ACT/trigger
# Restore normal microSD activity behaviour:
derek@rpi:~ $ echo mmc0 | sudo tee /sys/class/leds/ACT/trigger

Figure 2.14 lets you try these commands without a board, and shows which process opens the file in each form of the command.

Figure 2.14

The ACT LED through /sys/class/leds/, and who opens the file

Run the commands in order and watch the LED, the file list and the terminal.

ACT
Trigger: mmc0
LED: short flashes whenever the microSD card is read or written (the pattern drawn here is illustrative).
Driven by: the kernel's mmc0 trigger. User-space processes running: none.
0 s1 s2 s3 s
/sys/class/leds/ACT/ (attribute files)
  • -rw-r--r-- root rootbrightness= set by the trigger
  • -r--r--r-- root rootmax_brightness
  • -rw-r--r-- root roottrigger= [mmc0]

After boot the trigger is mmc0. Start by reading the trigger file.

Read
Set the trigger: echo NAME | sudo tee .../trigger
Set the LED: echo N | sudo tee .../brightness
Timer delays (ms): set the trigger to timer first
Terminal
derek@rpi:~ $ 

Not modelled: the PWR LED and differences between boards, the full trigger list, the oneshot and cpu triggers, brightness values above 1, and the heartbeat speeding up with CPU load.

Figure 2.14. The green ACT LED of the Raspberry Pi 4 driven through /sys/class/leds/ACT/. The kernel’s timer, heartbeat and mmc0 triggers blink it with no user-space process running, and a write to a root-owned sysfs file succeeds only when the process that opens the file is allowed to write it.

Try this:

  1. Click cat .../trigger, then the timer trigger button: delay_on and delay_off (500 ms each) appear in the file list. Click 100 / 100 and read 5 Hz at a 50% duty cycle, then 50 / 950 for 1 Hz at 5%.
  2. Click heartbeat: the delay files go, and the trace shows two short flashes in each period. Click 1 (on) and nothing lasting happens, because the trigger still controls the LED. Click none, then 1 (on) and 0 (off), and the LED follows.
  3. Switch to the second view, on why sudo echo > fails. Step through sudo echo >, echo | sudo tee and sudo sh -c '...' and note which process is marked as the one that opens the file. Then choose echo > with no sudo and tick the udev rule: the write works with no sudo (Section 2.7).

The figure does not model the PWR LED, the full trigger list, brightness values above 1, or the heartbeat speeding up with CPU load, and its mmc0 flicker is a fixed pattern.

A zero-code hardware shutdown button (gpio-shutdown)

Section titled “A zero-code hardware shutdown button (gpio-shutdown)”

In Section 2.2, we noted that you must never power off a Linux gateway by pulling the USB-C cable while the root file system is mounted read-write. The Raspberry Pi 5 has a power button on the PCB beside the status LED. A short press starts a clean systemd shutdown on Raspberry Pi OS Lite (the desktop asks for a second press to confirm).15

What about a headless Raspberry Pi 4 mounted inside an enclosure with no keyboard or SSH screen nearby? Older tutorials use a background shell or Python script that polls a GPIO pin and calls /sbin/shutdown -h now. That is no longer needed. Add one Devicetree overlay line to /boot/firmware/config.txt:

dtoverlay=gpio-shutdown,gpio_pin=17

The overlay’s defaults (active_low=1, gpio_pull=up) suit a button to ground.16 Wire a momentary push-button between Pin 11 (GPIO17) and Pin 9 (GND) and reboot. When a user presses the button:

  1. The kernel’s gpio-keys input driver detects the falling edge on GPIO17 and generates a standard Linux KEY_POWER event.16
  2. systemd-logind receives KEY_POWER and initiates a clean, orderly systemctl poweroff, stopping all services and unmounting the microSD card before halting the SoC. Why GPIO17 and not the overlay’s default pin, GPIO3 (Pin 5)? Many tutorials use GPIO3 because, with WAKE_ON_GPIO=1 in the bootloader configuration (the default, shown in Section 2.5), pulling GPIO3 low while a Pi 4 is halted starts the board again, so one button can both shut it down and wake it.15 However, GPIO3 is SCL1, the clock line of the I²C bus that we use for sensors in later chapters, and a pin cannot be both. With the button on GPIO17, restart a halted board by cycling its power.
Concept Match

Match the unit file line to what it does

Quiz
Select 0/1

Why does sudo echo none > /sys/class/leds/ACT/trigger fail with Permission denied?

Quiz
Select 0/1

You change RestartSec= in /etc/systemd/system/sensor-logger.service. What must you run before the next restart uses the new value?

Quiz
Select 0/2

Which of the following are reasons to prefer a systemd timer to a cron job on a gateway?

  1. D. Molloy, Exploring Raspberry Pi: Interfacing to the Real World with Embedded Linux, Wiley, 2016, ch. 3, “Exploring Embedded Linux Systems.” ISBN 978-1-119-18868-1. ↩

  2. systemd project, systemd(1): systemd system and service manager, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.html. ↩ ↩2 ↩3

  3. systemd project, runlevel(8), Table 1, “Mapping between runlevels and systemd targets”. Available: man7.org/linux/man-pages/man8/runlevel.8.html. ↩

  4. systemd project, systemctl(1): control the systemd system and service manager, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemctl.html. ↩ ↩2

  5. US National Highway Traffic Safety Administration, Federal Motor Vehicle Safety Standard No. 111: Rear visibility, 49 CFR 571.111, S5.5.3 (response time), required for all new light vehicles from 1 May 2018. Available: ecfr.gov. ↩

  6. systemd project, systemd-analyze(1): analyze and debug system manager, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd-analyze.html. ↩

  7. systemd project, journalctl(1): print log entries from the systemd journal, systemd 262. Available: freedesktop.org/software/systemd/man/latest/journalctl.html. ↩

  8. systemd project, systemd.service(5): service unit configuration, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.service.html. ↩ ↩2

  9. systemd project, systemd.unit(5): unit configuration, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.unit.html. ↩

  10. Linux man-pages project, setbuf(3): stream buffering operations, man-pages 6.19. Available: man7.org/linux/man-pages/man3/setbuf.3.html. ↩

  11. systemd project, systemd-system.conf(5), DefaultStartLimitIntervalSec= and DefaultStartLimitBurst=, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd-system.conf.html. ↩

  12. systemd project, systemd.timer(5): timer unit configuration, systemd 262. Available: freedesktop.org/software/systemd/man/latest/systemd.timer.html. ↩

  13. Debian, cron(8): daemon to execute scheduled commands, cron 3.0pl1, Debian trixie. Available: manpages.debian.org/trixie/cron/cron.8.en.html. ↩ ↩2

  14. The Linux Kernel documentation, LED handling under Linux. Available: docs.kernel.org/leds/leds-class.html. ↩ ↩2

  15. Raspberry Pi Ltd, Raspberry Pi hardware, sections “Power button” and “Raspberry Pi bootloader configuration” (WAKE_ON_GPIO). Available: raspberrypi.com/documentation/computers/raspberry-pi.html. ↩ ↩2

  16. Raspberry Pi Ltd, Device Tree overlays README, entry gpio-shutdown, raspberrypi/firmware repository. Available: github.com/raspberrypi/firmware/blob/master/boot/overlays/README. ↩ ↩2