Uptime Is Not Stability: Why Firmware Crashes on Day 21

What this is. Opinion + Experience + Fact (30% opinion · 40% experience · 30% fact). Written in collaboration with AI — I discuss, I do not outsource.

Why some embedded devices run perfectly for weeks and then crash for no visible reason — and why long-life reliability is a design decision, not a bug fix. The short version: a whole class of failures grows silently with uptime (memory leaks, heap fragmentation, counter overflow), so the fix is to design for the long run and watch the trend, not the snapshot.

Every embedded engineer eventually meets this device. It runs perfectly on the bench, passes every test, ships, and works flawlessly for three weeks. Then, with no new input and nothing obviously changed, it crashes. Reboot it and the clock starts again — fine for another few weeks, then the same fall. Nothing "happened" on day 21; the cause was there from the beginning, growing quietly the entire time.

These are the failures uptime reveals, and they're a different animal from a normal bug. A normal bug is a wrong answer you can reproduce. This is a device that is slowly using itself up. Here's why it happens and how to design so it doesn't.

---

1. Some bugs need time to grow

A class of problems is invisible at small time scales and inevitable at large ones. A slow memory leak that loses a few bytes per cycle is nothing in a demo and fatal after a million cycles. A counter or timestamp that increments steadily will, eventually, overflow — on a schedule you can calculate but will never see in testing. State accumulates: caches grow, fragmentation spreads, a queue that's almost always drained occasionally isn't. None of these throw an error early; they just build until something tips over.

That's what makes them dangerous: they don't fail on your bench, they fail on a customer's device weeks later, where you can't watch. (It's the same reason the bench is not the field — the bench doesn't run long enough.)

▸ First principle. A bug that grows with time is invisible until enough time has passed.

2. Heap fragmentation is the classic culprit

The most famous of these is heap fragmentation, and it fools people because it looks like there's plenty of memory right up until the crash. Here's the mechanism: every time firmware allocates and frees blocks of different sizes over a long run, the free memory gets chopped into scattered fragments. Eventually the program asks for one contiguous block big enough for something — and even though the total free memory is more than enough, no single gap is large enough to satisfy it. The allocation fails, and the device falls over.

This is external fragmentation, and it's why "free memory looks healthy" is not the same as "the next allocation will succeed." It also explains the maddening non-determinism: the exact moment it tips depends on the history of allocations, which is why two identical devices can crash days apart. The deeper lesson is that on a device meant to run for years, how you allocate memory matters more than how much you have.

▸ First principle. On a device that runs for years, how you allocate memory matters more than how much you have.

3. Design for the long run from day one

The good news is that these failures are largely designed out, not debugged out. In long-lived firmware, the standard defenses are to avoid ad-hoc dynamic allocation in the steady-state loop — prefer static allocation or fixed-size memory pools whose behavior is bounded and predictable — and to bound every buffer, queue, and cache so nothing can grow without limit. (This is exactly why safety-critical coding standards discourage or forbid heap allocation after startup.)

The second defense is visibility: you catch a slow drift by watching a number move across days, not by glancing at a snapshot. Free-memory low-water mark, largest allocatable block, task stack high-water marks — trend them, and a leak or creeping fragmentation shows up as a line sloping the wrong way long before it crashes. That trend-over-time visibility is exactly what on-device observability is for, and it's part of why I build EmbedIQ, my open framework for embedded and edge.

▸ First principle. Long-life reliability is a design choice made on day one, not a patch after day 21.

Sources

The mechanisms here — memory leaks, external heap fragmentation, counter/timestamp overflow, and the preference for static allocation and bounded buffers in long-running/safety-critical firmware — are standard embedded-engineering practice. On embedded memory and fragmentation, see the Memfault Interrupt blog on embedded memory. This piece is method and lived experience; "day 21" and "three weeks" are illustrative of the pattern, not measured claims.

FAQ

Why does an embedded device run fine for weeks and then crash?
Because a class of problems grows with uptime rather than triggering immediately: a slow memory leak, heap fragmentation, a counter creeping toward overflow, or unbounded state. Each is invisible in short testing and only tips over after enough time — usually in the field, not on the bench.

What is heap fragmentation and why does it crash devices?
Over a long run of allocating and freeing different-sized blocks, free memory gets split into scattered fragments. Eventually a request for one big-enough contiguous block fails even though total free memory is ample — no single gap is large enough. That failed allocation crashes the device, which is why free memory can look fine right before the crash.

How do you prevent long-uptime crashes in firmware?
Design for it: prefer static allocation or fixed-size memory pools over ad-hoc dynamic allocation in the steady state, bound every buffer/queue/cache, and watch trends over time (free-memory low-water mark, largest allocatable block, stack high-water marks) so a slow drift is visible days before it becomes a crash.

Is heap fragmentation the same as a memory leak?
No. A leak loses memory permanently; fragmentation keeps the memory but scatters it so it can't be used in one piece. Both are time-dependent and both crash long-running devices, but the fixes differ — leaks need the loss found and stopped, fragmentation needs allocation strategy changed (pools, static allocation, bounded lifetimes).

This is an embedded field note. If you're building something that has to run for years without a reboot, the longer story and how I work live on my profile. — Ritesh | ritzylab.com