Skip to content

2.8 Storage, file systems, and flash wear

In UNIX and Linux, “everything is a file”. Whether your C++ gateway application is reading a configuration JSON on an ext4 partition, writing to an I²C bus controller (/dev/i2c-1), inspecting the kernel’s live Devicetree (/proc/device-tree/), or reading the SoC die temperature (/sys/class/thermal/thermal_zone0/temp), it uses the exact same Virtual File System (VFS) calls: open(), read(), write(), ioctl(), and close().1

However, an embedded Linux gateway differs sharply from a cloud server in what backs its storage. Most gateways boot from NAND flash memory: a removable microSD card, soldered eMMC, or an M.2 NVMe drive. If you treat a consumer microSD card like a server hard drive, writing high-rate sensor logs or database updates every second, the card’s flash cells can wear out and fail early.


The Linux directory hierarchy (FHS) on a modern gateway

Section titled “The Linux directory hierarchy (FHS) on a modern gateway”

Table 2.10 maps the top-level directories you will navigate on Raspberry Pi OS (which follows the modern Debian merged-/usr layout, where /bin, /sbin, and /lib are symbolic links into /usr/bin, /usr/sbin, and /usr/lib).2

Table 2.10. Essential directories in the Linux Filesystem Hierarchy Standard (FHS) on Raspberry Pi OS.3

DirectoryStorage BackingEngineering Role on a Connected Gateway
/ext4 (mmcblk0p2)The root directory at the top of the Virtual File System tree.
/boot/firmware/vfat / FAT32 (mmcblk0p1)Bootloader configuration (config.txt, cmdline.txt), kernel images (kernel8.img, kernel_2712.img), .dtb files, and overlays/.
/dev/devtmpfs (RAM)Character and block device nodes populated dynamically by systemd-udevd (/dev/gpiochip0, /dev/i2c-1, /dev/mmcblk0).
/etc/ext4System-wide configuration files: /etc/fstab, /etc/systemd/system/, /etc/udev/rules.d/, /etc/mosquitto/.
/home/ext4User home directories (/home/derek). (Note: the superuser’s home is /root, not /home/root, on FHS distributions).
/proc/proc (Virtual RAM)Live window into the running Linux kernel and processes (/proc/cpuinfo, /proc/meminfo, /proc/cmdline, /proc/device-tree/). Occupies 0 bytes on flash.
/run/tmpfs (RAM)Volatile runtime data since boot: PID files, UNIX domain sockets, NetworkManager and systemd state. Cleared automatically on reboot.
/sys/sysfs (Virtual RAM)Structured kernel device, bus, and driver model (/sys/class/leds/, /sys/class/thermal/, /sys/bus/i2c/). Occupies 0 bytes on flash.
/tmp/tmpfs (RAM)Temporary scratch files (1777 sticky bit). On Debian 13 (Trixie), /tmp is a tmpfs in RAM by default,4 so writing to it causes no microSD wear. On Bookworm it was a directory on the ext4 root.
/usr/ext4User-space binaries (/usr/bin), system daemons (/usr/sbin), libraries (/usr/lib), and locally compiled gateway software (/usr/local/bin).
/var/ext4 (or tmpfs)Variable data modified during operation: /var/log/ (system journals), /var/lib/ (persistent service state, databases), /var/cache/.

Section titled “Inodes, hard links, and symbolic (soft) links”

How does the ext4 file system actually store a file? Every file or directory on an ext4 partition is represented by a numbered metadata structure called an Index Node (inode).

An inode stores the file’s owner (UID/GID), permission bits (0644), exact size in bytes, creation/modification timestamps, and pointers to the physical data blocks on the flash media: everything about the file except its name.5 A directory is a lookup table mapping human-readable filenames to inode numbers (ls -i displays the inode number of any file).

Understanding inodes makes the difference between a hard link (ln) and a symbolic (soft) link (ln -s) clear:

Figure 2.17

Hard links and symbolic links

Click any block or link to see what it does.

Directory /tmp
Inodes
Data
derek@rpi:/tmp $ ls -li cal*
41028 -rw-r--r-- 2 derek derek 23 Nov  7 15:00 cal.txt
41028 -rw-r--r-- 2 derek derek 23 Nov  7 15:00 cal_hard.txt
41099 lrwxrwxrwx 1 derek derek  7 Nov  7 15:00 cal_soft.txt -> cal.txt

Click any box or arrow. Then switch to 'After rm cal.txt' to see which link survives.

Figure 2.17. Hard links share the exact same underlying inode and data blocks, whereas a symbolic (soft) link is a separate pointer file storing a target path string.

Let us verify this behaviour with a quick experiment in /tmp:

Terminal window
derek@rpi:/tmp $ echo "Gateway calibration v1" > cal.txt
derek@rpi:/tmp $ ln cal.txt cal_hard.txt # Create a hard link
derek@rpi:/tmp $ ln -s cal.txt cal_soft.txt # Create a symbolic (soft) link
derek@rpi:/tmp $ ls -li cal*
41028 -rw-r--r-- 2 derek derek 23 Nov 7 15:00 cal.txt
41028 -rw-r--r-- 2 derek derek 23 Nov 7 15:00 cal_hard.txt
41099 lrwxrwxrwx 1 derek derek 7 Nov 7 15:00 cal_soft.txt -> cal.txt

Notice three critical details:

  1. cal.txt and cal_hard.txt share the exact same inode number (41028), and the link count column after the permissions has incremented from 1 to 2. Neither is the “original” or the “copy”: a normal filename is itself a hard link to an inode.
  2. If you delete or rename cal.txt (rm cal.txt), cal_soft.txt breaks immediately (it becomes a dangling symlink pointing to a non-existent path), whereas cal_hard.txt continues to work intact because the kernel only frees an inode and its data blocks when its hard-link count drops to 0 (and no running process holds an open file descriptor to it).6
  3. Because inode numbers are unique only within a single file system, hard links cannot cross file-system mounts (e.g., from /boot/firmware to /home) and cannot link to directories, whereas symbolic links (ln -s) can point anywhere across the system.7

Block devices, mounting, and /etc/fstab (UUID and nofail)

Section titled “Block devices, mounting, and /etc/fstab (UUID and nofail)”

Use lsblk and df -Th to inspect your gateway’s block devices and mounted file systems:

Terminal window
derek@rpi:~ $ lsblk -f
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
mmcblk0
├─mmcblk0p1 vfat bootfs 5A4C-9E12 442.1M 13% /boot/firmware
└─mmcblk0p2 ext4 rootfs 8d4c21a0-3f11-4b92-9e81-0123456789ab 24.8G 9% /

(Note on GB vs GiB: Storage manufacturers label a 32 GB microSD card in decimal gigabytes (32 × 109 bytes), whereas Linux tools such as lsblk and df -h report binary gibibytes, 1 GiB = 230 = 1,073,741,824 bytes.8 That is why a new 32 GB card shows a total capacity of 29.8 GiB.)

Safely mounting external USB or NVMe storage in /etc/fstab

Section titled “Safely mounting external USB or NVMe storage in /etc/fstab”

Suppose your gateway needs an external USB flash drive (or an M.2 NVMe SSD) formatted as ext4 (sudo mkfs.ext4 -L telemetry /dev/sda1) mounted automatically at /mnt/telemetry on every boot.

Two mistakes in /etc/fstab frequently leave headless gateways unreachable in the field:

  1. Using /dev/sda1 instead of UUID=: Kernel device names like /dev/sda1 and /dev/sdb1 are assigned in the order USB devices happen to enumerate at boot. Instead, find the file system’s Universally Unique Identifier with sudo blkid /dev/sda1 and mount by UUID=....9 The UUID is stored in the file system itself and stays the same until you reformat the partition.
  2. Omitting the nofail mount option: By default, if a partition listed in /etc/fstab is unplugged or fails to spin up at boot, systemd halts the boot sequence and drops into emergency.target on the local console, so sshd never starts and your headless gateway is unreachable over the network.10 Always include nofail on external data drives so the gateway still boots and starts SSH even if the USB drive was unplugged:
# /etc/fstab entry for an external telemetry drive:
UUID=a1b2c3d4-e5f6-7890-abcd-ef0123456789 /mnt/telemetry ext4 defaults,noatime,nofail 0 2

Notice the noatime option on the root partition in Raspberry Pi OS’s default /etc/fstab. Under strict POSIX rules, every time a process only reads a file (cat, loading a shared library), the kernel updates the file’s atime (last-access timestamp) in its inode, so a read causes a flash write. Linux’s default mount behaviour, relatime, already removes most of these writes: it updates atime only when the old value is earlier than the last modification or more than 24 hours old.11 noatime removes the remainder, and very few programs need access times at all.


NAND flash physics, microSD wear, and how to protect your gateway

Section titled “NAND flash physics, microSD wear, and how to protect your gateway”

Why do microSD cards wear out under continuous logging? Both microSD cards and soldered eMMC chips use NAND flash memory, where bits are stored as electrical charge trapped on an isolated floating gate or charge-trap layer inside a transistor:

  1. Pages are programmed, blocks are erased: NAND flash is read and programmed in small pages (typically 4 KB to 16 KB), but a page cannot be overwritten in place.12 It must be erased first, and erasure works only on a whole erase block (typically 256 KB to 4 MB). The card’s controller therefore writes changed data to a fresh page somewhere else and marks the old page as stale in its translation table. Later, garbage collection copies the pages that are still valid out of a mostly stale block and erases that block. Appending a 100-byte line to a log programs at least one whole page (plus pages for the file-system metadata and journal), and the copying during garbage collection adds more. The ratio of the bytes programmed to the bytes that the host asked to write is the write amplification.13
  2. Finite Program/Erase (P/E) endurance across cell generations: Each high-voltage erase pulse degrades the thin oxide insulating layer around the cell.12 As flash manufacturers pack more bits (voltage levels) into a single transistor cell, the voltage margin between states shrinks and endurance drops sharply (Table 2.11).

Table 2.11. NAND flash cell architectures and typical Program/Erase (P/E) cycle endurance.

NAND Cell TypeBits per CellVoltage States per CellTypical P/E Cycle EnduranceTypical Deployment
SLC (Single-Level Cell)1 bit2 states60,000 – 100,000 cycles12High-reliability industrial eMMC and aerospace flash.
pSLC (pseudo-SLC)1 bit stored in an MLC or TLC cell2 states20,000 – 30,000 cyclesIndustrial microSD cards and eMMC, at a half or a third of the cell’s normal capacity.
MLC (Multi-Level Cell)2 bits4 states3,000 – 10,000 cycles12Older industrial cards and SSDs; now uncommon in consumer cards.
TLC (Triple-Level Cell)3 bits8 states1,000 – 3,000 cyclesStandard consumer A1/A2 microSD cards and mainstream NVMe SSDs.
QLC (Quad-Level Cell)4 bits16 states300 – 1,000 cyclesHigh-capacity, low-cost consumer flash (unsuitable for write-heavy logging).

Four practical steps to prevent microSD failure on a gateway

Section titled “Four practical steps to prevent microSD failure on a gateway”
  1. Choose a High-Endurance card and leave free space: Flash controllers perform dynamic and static wear levelling, spreading writes across the free blocks on the card. A 32 GB High-Endurance card that is only 20% full has ~25 GB of free blocks to rotate writes across, and lasts much longer than an almost-full 8 GB consumer card. (The Application Performance Classes A1 and A2 are random-I/O ratings, measured in IOPS.14 They say nothing about endurance.)
  2. Keep volatile files in tmpfs (in RAM): Place scratch files and high-rate transient state in RAM-backed tmpfs file systems. /run is a tmpfs, and on Trixie /tmp is too (check with findmnt /tmp). On Bookworm, or for a scratch directory of your own, add one line to /etc/fstab (a tmpfs uses physical RAM only for the files stored in it, up to the size= cap):15
    tmpfs /tmp tmpfs defaults,noatime,nosuid,nodev,size=100M 0 0
  3. Keep systemd-journald logs in volatile RAM or use log2ram: In /etc/systemd/journald.conf, setting Storage=volatile and RuntimeMaxUse=32M keeps all journalctl system logs in /run/log/journal (RAM) so the journal causes no flash writes.16 The cost is that the logs are lost at every reboot or power cut, including the messages that would explain a crash, so send the events that matter off the board.
  4. Enable a Read-Only Root File System (OverlayFS) for field deployment: Once a gateway is ready for deployment in a harsh environment where power may be cut at any instant, open sudo raspi-config → Performance Options → Overlay File System.17 This configures the Linux kernel’s overlayfs, mounting your microSD ext4 root partition read-only (lowerdir) and placing a tmpfs RAM layer (upperdir) on top.18 Every change is made in RAM, so no write to the card is in progress when the power is cut and the root file system cannot be left half-written. The costs are that every change since boot is lost, that the RAM layer can fill, and that to update the system you must disable the overlay, reboot, make the change, and enable it again. The card can still fail for other reasons.

Figure 2.18 shows where each write goes with the overlay off and on.

Figure 2.18

A read-only root with OverlayFS: where each write goes

Run the operations with the overlay off, then on. Changing the setting reboots the Pi.

What ls and every process see at /
Upper layer: not mounted while the overlay is off

Writes go straight to the SD card.

ext4 on the microSD card, mounted read-write
  • /etc/gateway.confinterval_s=5212 B
  • /etc/hostname4 B
  • /usr/local/bin/sensor-logger68 KiB
  • /var/log/sensor.log48 KiB
File operationsClick a file in the top panel to choose what Delete removes.
Then
Written to the SD card
0 B
RAM used by the upper layer
not mounted
Boots since the start: 0
Overlay off: / is the ext4 partition, mounted read-write
Every change is written to the SD card. Try each operation, then switch the overlay on and repeat them.
derek@rpi:~ $ ls /etc/hostname /etc/gateway.conf /var/log/sensor.log
/etc/gateway.conf  /etc/hostname  /var/log/sensor.log

/var/log/sensor.log (48 KiB)

Stored on the ext4 partition of the SD card. With the overlay off, reads and writes both go to the card.

Sizes are scaled down: the upper layer is capped at 256 KiB here, where a real tmpfs defaults to half of the RAM. Counters are in 4 KiB blocks, with one extra block of ext4 metadata and journal for each operation on the card. Not modelled: the copy-up of metadata (owner, mode, timestamps), opaque directories, the initramfs that sets the overlay up, the page cache, and wear levelling inside the card.

Figure 2.18. A read-only root with OverlayFS on the Raspberry Pi gateway. With the overlay on, every change goes to a tmpfs in RAM above the read-only ext4 partition on the microSD card, so nothing is written to the card and nothing survives a reboot or power cut.

Try this:

  1. With the overlay off, click Append to the log three times and then Cut the power during a write. The SD counter rises with each append, and after the power cut the log ends in an incomplete record.
  2. Switch to Overlay on (this reboots the Pi), click Edit gateway.conf, then select /etc/hostname in the top panel and click Delete. The configuration file is copied up to RAM, a whiteout hides the hostname file, and the SD counter does not change. Click Reboot and both changes are gone.
  3. With the overlay on, click Append to the log six times. The first append copies the whole log into RAM, and the sixth fails with No space left on device. Select sensor.log and delete it, and the RAM is freed.

The sizes in the figure are scaled down so that the RAM layer fills in a few clicks (a real tmpfs defaults to half of RAM).15 It does not model the copy-up of metadata, the overlay work directory, the page cache, or wear levelling inside the card.

The lab below goes further than the notes above. Tab 1 is a small flash translation layer: run it with no wear levelling, then with dynamic and static wear levelling, at different card fills, and compare how many writes the card takes before it runs out of spare blocks. Tab 2 shows why each extra bit per cell costs endurance, and Tab 3 estimates how long a card lasts under a logging service, including the cost of calling fsync() on every record and the effect of the four steps above.

Gateway storage

NAND flash wear on a gateway

Card fill (cold data)
Workload
Wear levelling
Cell type (endurance ÷ 50)
00/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/2000/20
erased page cold data (static) hot data stale copy block being written last block collected erases / endurance
Host page writes
0
Flash page programs
0
host writes + pages copied by GC
Write amplification
–
Erase count min / avg / max
0 / 0.0 / 0
Worn-out blocks
0 of 32
Average life used
0 %
No wear levelling: the controller always reuses the lowest-numbered erased block, so the same few blocks take almost every erase while the rest of the card stays new. Over-provisioning: 32 physical blocks hold 28 blocks of logical data, and the cold data fills 50 % of that.

Simplified models. What this does not model: read disturb, data retention over time, SLC caching, TRIM, the file system's own block allocation, and the real controller in a microSD card, whose algorithms are not published.

Benchmarking storage speed (hdparm and NVMe on Pi 5)

Section titled “Benchmarking storage speed (hdparm and NVMe on Pi 5)”

You can measure sequential read throughput across any block device using sudo hdparm -tT (-t measures raw buffered disk reads; -T measures the Linux kernel’s RAM page cache):19

Terminal window
# On a Raspberry Pi 4 with an A2 microSD card (DDR50 bus limit, about 40 to 45 MB/s):
derek@rpi:~ $ sudo hdparm -tT /dev/mmcblk0
/dev/mmcblk0:
Timing cached reads: 3140 MB in 2.00 seconds = 1571.12 MB/sec
Timing buffered disk reads: 130 MB in 3.02 seconds = 43.05 MB/sec
# On a Raspberry Pi 5 with an M.2 NVMe SSD over PCIe (Gen 3 enabled):
derek@rpi:~ $ sudo hdparm -tT /dev/nvme0n1
/dev/nvme0n1:
Timing cached reads: 6820 MB in 2.00 seconds = 3412.40 MB/sec
Timing buffered disk reads: 2410 MB in 3.00 seconds = 803.15 MB/sec

The Pi 5’s PCIe port runs at Gen 2 by default, which gives about 400–450 MB/s, roughly ten times the Pi 4’s microSD interface. Adding dtparam=pciex1_gen=3 to config.txt selects Gen 3, which is not officially certified on the Pi 5 and gave the figure above.20 The improvement in random 4K IOPS and in endurance is larger again, which matters more than sequential speed for a gateway that runs a database.


Rescuing a gateway microSD card that will not boot

Section titled “Rescuing a gateway microSD card that will not boot”

Sooner or later, every embedded engineer makes a typo in /etc/fstab, /boot/firmware/cmdline.txt, or /etc/sudoers that stops a board from booting. Because the gateway’s storage is a removable microSD card, rescuing it takes a few minutes:

  1. If the mistake was in /boot/firmware/config.txt or cmdline.txt: Plug the microSD card into any Windows, macOS, or Linux laptop. The first partition (bootfs) is standard FAT32 and opens immediately in your file manager (Notepad / VS Code). Fix the line, eject safely, and boot.
  2. If the mistake was in /etc/fstab or /etc/sudoers on the ext4 root partition: Plug the unbootable microSD card into a USB microSD card reader and plug that USB reader into a working Raspberry Pi (or any Linux PC). Mount the second partition (/dev/sda2) and fix the file:
    Terminal window
    derek@rpi:~ $ sudo mkdir -p /mnt/rescue
    derek@rpi:~ $ sudo mount /dev/sda2 /mnt/rescue
    derek@rpi:~ $ sudo nano /mnt/rescue/etc/fstab # Fix the broken line!
    derek@rpi:~ $ sync
    derek@rpi:~ $ sudo umount /mnt/rescue
    For a broken sudoers file, use sudo visudo -f /mnt/rescue/etc/sudoers, which checks the syntax before it saves.21 Move the card back to your gateway and it will boot.
Concept Match

Match the /etc/fstab setting to its purpose

Quiz
Select 0/1

A file cal.txt has a hard link, cal_hard.txt, and a symbolic link, cal_soft.txt. You run rm cal.txt. What happens?

Quiz
Select 0/1

Why does appending one 100-byte line to a log file cause far more than 100 bytes of flash wear?

Quiz
Select 0/3

Which of the following reduce the wear on the microSD card of a logging gateway?

  1. D. Molloy, Exploring Raspberry Pi: Interfacing to the Real World with Embedded Linux. Indianapolis, IN, USA: Wiley, 2016, ch. 3, ISBN 978-1-119-18868-1. ↩

  2. Debian Project, UsrMerge, Debian Wiki. Available: wiki.debian.org/UsrMerge. ↩

  3. LSB Workgroup, The Linux Foundation, Filesystem Hierarchy Standard, version 3.0, Mar. 2015. Available: refspecs.linuxfoundation.org/FHS_3.0. ↩

  4. Debian Project, “The temporary-files directory /tmp is now stored in a tmpfs,” in Release notes for Debian 13 (trixie), sec. 5.1.6. Available: debian.org/releases/trixie/release-notes. ↩

  5. The Linux man-pages project, inode(7): Linux manual page. Available: man7.org/linux/man-pages/man7/inode.7.html. ↩

  6. The Linux man-pages project, unlink(2): Linux manual page. Available: man7.org/linux/man-pages/man2/unlink.2.html. ↩

  7. The Linux man-pages project, link(2): Linux manual page. Available: man7.org/linux/man-pages/man2/link.2.html. ↩

  8. National Institute of Standards and Technology, Definitions of the SI units: The binary prefixes (IEC 60027-2). Available: physics.nist.gov/cuu/Units/binary.html. ↩

  9. The Linux man-pages project, fstab(5): Linux manual page. Available: man7.org/linux/man-pages/man5/fstab.5.html. ↩

  10. The systemd project, systemd.mount(5), systemd manual pages. Available: freedesktop.org/software/systemd/man/latest/systemd.mount.html. ↩

  11. The Linux man-pages project, mount(8): Linux manual page. Available: man7.org/linux/man-pages/man8/mount.8.html. ↩

  12. L. M. Grupp, A. M. Caulfield, J. Coburn, S. Swanson, E. Yaakobi, P. H. Siegel, and J. K. Wolf, “Characterizing Flash Memory: Anomalies, Observations, and Applications,” in Proc. 42nd Annu. IEEE/ACM Int. Symp. Microarchitecture (MICRO-42), New York, NY, USA, Dec. 2009, pp. 24–33, doi: 10.1145/1669112.1669118. ↩ ↩2 ↩3 ↩4

  13. X.-Y. Hu, E. Eleftheriou, R. Haas, I. Iliadis, and R. Pletka, “Write Amplification Analysis in Flash-Based Solid State Drives,” in Proc. SYSTOR 2009: The Israeli Experimental Systems Conference, May 2009, pp. 1–9, doi: 10.1145/1534530.1534544. ↩

  14. SD Association, Mobile Device Innovations: Application Performance Class Expansion and New Low Voltage Signaling for SD Memory Cards, white paper, Feb. 2017. Available: sdcard.org. ↩

  15. The Linux Kernel documentation, Tmpfs. Available: docs.kernel.org/filesystems/tmpfs.html. ↩ ↩2

  16. The systemd project, journald.conf(5), systemd manual pages. Available: freedesktop.org/software/systemd/man/latest/journald.conf.html. ↩

  17. Raspberry Pi Ltd, Configuration, Raspberry Pi documentation (raspi-config, Performance Options, Overlay File System). Available: raspberrypi.com/documentation/computers/configuration.html. ↩

  18. The Linux Kernel documentation, Overlay Filesystem. Available: docs.kernel.org/filesystems/overlayfs.html. ↩

  19. The Linux man-pages project, hdparm(8): Linux manual page. Available: man7.org/linux/man-pages/man8/hdparm.8.html. ↩

  20. Raspberry Pi Ltd, Raspberry Pi hardware, Raspberry Pi documentation. Available: raspberrypi.com/documentation/computers/raspberry-pi.html. ↩

  21. The Sudo project, visudo(8) manual. Available: sudo.ws/docs/man/visudo.man. ↩