What this is. Opinion + Experience + Fact (30% opinion · 40% experience · 30% fact). Written in collaboration with AI — I discuss, I do not outsource.

Why the logs you most need — the ones from the moment a device crashed — are usually the ones that disappear, and how crash-safe logging keeps them. The short version: a crash is a reset, RAM doesn't survive a reset, so the fault handler has to record why before the chip reboots, into storage that persists.

A RAM-only log dies on reset; a fault-handler write survives it On the left, a crash resets the chip and wipes RAM, so a log held only in RAM is gone. On the right, the fault handler writes the essential failure record to persistent storage before the reset, so on the next boot the evidence can be read back. A crash is a reset — what survives it? Log in RAM only RAM: recent events (volatile) reset RAM wiped — gone Fault handler writes before the reset fault handler: what faulted + where persistent store (flash / noinit) survives the reset next boot: read it back the evidence is there

Here's a maddening loop every embedded engineer knows. A device crashes and reboots. It comes back up looking perfectly healthy. You open the logs to see what happened at the moment it died — and there's nothing, because those logs were sitting in RAM (or an unflushed buffer), and the crash wiped RAM on the way down. The single most valuable evidence, gone at exactly the moment it existed.

This is the crash-safe logging problem, and it's the difference between a field failure you diagnose once and one that haunts you for months. Here's how to keep the evidence.

1. The crash takes the evidence with it

Most logging assumes a cooperative shutdown: you write to a buffer, and eventually it's flushed to storage or sent off the device. A crash is the opposite of cooperative. A hard fault, a watchdog reset, a brown-out — none of them flush your buffer or give you a clean exit. Anything that only lived in volatile RAM is gone the instant the core resets. So "I'll check the logs" quietly becomes "there are no logs for the part that actually broke," which is the worst possible time to have none.

▸ First principle. A log that only lives in RAM won't survive the thing you most need it for.

2. Capture the crash before the reset

The fix is to do a little work in the fault handler itself — the code that runs the moment a fault occurs, before the device resets. Its one job is to record the minimal truth: what faulted, where the processor was when it happened (on Cortex-M, the fault status registers plus the stacked program counter and link register), and a few recent breadcrumbs. Write that to somewhere that survives a reset — a reserved flash region, backup registers, or a RAM section your startup code is told not to wipe. Then let the device reboot, and on the next boot, read that record back and report it. A "coredump on fault" is the fuller version of the same idea.

▸ First principle. The fault handler's one job is to record why, before the chip forgets.

3. Keep a small persistent breadcrumb trail

Beyond the crash snapshot, a tiny always-running trail turns mysteries into stories. A small ring buffer of recent significant events — kept in non-volatile storage, or in a "noinit" RAM region the startup code deliberately doesn't clear on a warm reset — means that after any unexpected reboot you can read the last handful of things the device did before it went down. The critical point: this has to be running before the failure. You cannot add logging after the crash; the trail either existed at the moment it broke, or it didn't. That always-on, on-device recording is exactly what EmbedIQ, my open framework for embedded and edge, is built around.

▸ First principle. The evidence has to be recorded before the failure, not after.

Sources

The mechanisms here — RAM loss on reset, fault-handler crash capture (fault status registers plus stacked PC/LR on Cortex-M), coredumps, persistent "noinit"/backup-RAM regions, and non-volatile ring-buffer logging — are standard embedded practice. On embedded coredumps and crash capture, see the Memfault Interrupt writeup on Cortex-M faults. This piece is method and lived experience; it contains no product performance figures.

FAQ

Why do logs disappear when an embedded device crashes?
Because a crash is a reset, and anything held only in RAM or an unflushed buffer is lost when the core resets. Normal logging assumes a cooperative flush that a hard fault, watchdog reset, or brown-out never gives you — so the logs from the moment of failure vanish exactly when you need them.

What is crash-safe logging?
It's capturing the essential failure information before the device resets and storing it somewhere non-volatile: in the fault handler, record what faulted and where (on Cortex-M, the fault registers and stacked PC/LR) plus recent breadcrumbs, write it to flash or a reset-persistent region, and read it back on the next boot.

How do you keep logs across a reboot?
Use storage that survives a reset: a reserved flash area, backup registers, or a RAM section the startup code doesn't zero on a warm reset ("noinit"). Keep a small ring buffer of recent events there so an unexpected reboot leaves a readable trail.

Why must logging be running before the crash?
Because you can't retroactively record what already happened. If the breadcrumb trail and fault capture weren't running at the moment of failure, there's nothing to read afterward. Crash-safe logging has to be designed in and always on, not added after the first mystery reboot.

This is an embedded field note. If a device is rebooting in the field and taking its evidence with it, the longer story and how I work live on my profile. — Ritesh | ritzylab.com