The Bench Is Not the Field: Bugs You Can't Reproduce

What this is. Opinion + Experience + Fact (25% opinion · 45% experience · 30% fact). Written in collaboration with AI — I discuss, I do not outsource.

Why some firmware bugs pass every test on the bench and then fail at the customer — and what to do when you can't reproduce the failure. The short version: the bench quietly removes the three things that cause field failures — real power, real time, and real scale. If you can't bring the bug to the bench, make the field observable instead.

Every embedded engineer has met this bug. It passes every test on your desk. Then it fails at the customer's site — intermittently, unhelpfully, and never while you're watching. You can't reproduce it, so you can't fix it, and the ticket sits open while everyone quietly hopes it was a fluke. It usually isn't.

After taking a lot of products from bench to field over the years, I've stopped treating these as mysteries. The bench isn't lying at random; it's lying in three specific, predictable ways. Once you know them, "can't reproduce it" turns into "here's where to look."

---

1. The bench has clean power. The field doesn't.

Your development supply is regulated, filtered, and quiet. The field is none of those. Devices see brown-outs and voltage sag when a motor kicks in, electromagnetic interference from nearby radios and switching supplies, temperature swings, humidity, and vibration. A large share of the "impossible" bugs are simply the hardware being pushed outside the narrow, comfortable envelope the bench never leaves — a marginal reset line that only glitches when the rail dips, an ADC reading that's clean until an EMI burst, a sensor that misbehaves at 5 °C but not at room temperature.

This is why environmental and EMC testing exists as a discipline: the operating environment is part of the specification, not a footnote. When a failure only appears in the field, the environment is the first suspect, because it's the one variable your bench deliberately eliminates.

▸ First principle. A test that removes the environment removes the bug you're chasing.

2. The bench runs for minutes. The field runs for months.

The second lie is time. On your desk, a device lives for the length of a test — minutes, maybe an afternoon. In the field it runs continuously for weeks, months, years. A whole class of bugs is invisible until enough time has passed: a slow memory leak that only matters after millions of allocations, heap fragmentation that eventually can't satisfy a request, a counter or timestamp that overflows on day 21, a buffer that only overruns after a rare-but-inevitable sequence of events.

These aren't exotic. They're the ordinary consequence of code that was only ever observed young. You cannot demo your way into these failures — the device has to live long enough to break, and your bench session ends first. This is why soak testing and long-duration runs exist, and why "it ran all afternoon" is not the same claim as "it runs for a year."

▸ First principle. If the failure needs uptime, no amount of bench time reproduces it.

3. The bench sees one device. The field sees ten thousand.

The third lie is scale. On the bench you have one unit, or a handful. In production you have a fleet. A timing race with a one-in-ten-thousand chance per boot is effectively invisible on your desk — you'd have to reboot ten thousand times to see it once — and yet across a fleet of ten thousand devices it happens on roughly every boot cycle, somewhere. Rare, multiplied by scale, becomes routine.

The same logic applies to manufacturing spread, to edge-case sensor inputs, to network conditions, to user behavior. The bench samples one point in a distribution; the field samples the whole distribution, continuously. Some bugs simply are not a property of a unit — they're a property of the fleet — and no single-device test will ever surface them. (This is the same reason a handful of fleet-health numbers tells you more than any one device's logs.)

▸ First principle. Some bugs are a property of the fleet, not the unit.

4. If you can't bring the bug to the bench, make the field observable

Put the three lies together and a strategy falls out. You will never reliably reproduce a real-power, long-uptime, fleet-scale failure on your desk — so stop trying to drag the failure to the bench, and instead instrument the place where it actually happens. Make the field observable: capture what the device was doing at the moment it broke, on the device itself, so the next occurrence arrives carrying its own evidence instead of leaving you to guess.

This is the quiet backbone of EmbedIQ, the open (Apache-2.0) framework I've been building for embedded and edge: a way to see what a device actually did at runtime without hand-wiring instrumentation after the fact. It doesn't make the environment, the uptime, or the scale go away — nothing does — but it changes the debugging loop from "wait for it to happen again and hope you're watching" to "read what happened." When a bug only lives in the field, observability in the field is the fix. (It's the same instinct behind knowing exactly what you shipped before you go chasing a field report.)

▸ First principle. When a failure only lives in the field, the answer isn't a better bench — it's a field you can read.

Sources

The mechanisms here are standard embedded-engineering knowledge: environmental and EMC effects on hardware behavior (electromagnetic interference, brown-out, thermal), time-dependent faults (memory leaks, heap fragmentation, counter overflow), and probability-times-scale failures across fleets. On heap fragmentation and long-uptime memory behavior in embedded systems, see the Memfault Interrupt blog on embedded memory. This piece is method and lived experience; it contains no product performance figures, and "ten thousand" and "day 21" are illustrative, not measured claims.

FAQ

Why does a firmware bug pass on the bench but fail in the field?
Because the bench removes the three things that cause field failures: real power (brown-outs, sag, EMI, temperature), real time (weeks or months of continuous uptime), and real scale (a whole fleet instead of one device). A bug that depends on any of those is invisible on a desk and routine in production.

How do you debug a failure you can't reproduce?
Stop trying to reproduce it on the bench and make the field observable instead. Capture what the device was doing at the moment it failed, on the device itself, so the next occurrence brings its own evidence. You move from guessing to reading.

What field conditions cause bugs that don't show on the bench?
Unregulated or noisy power and brown-outs, electromagnetic interference, temperature and humidity extremes, vibration, long continuous uptime (memory leaks, fragmentation, counter overflow), and fleet-scale probability effects where a rare timing race becomes common across many devices.

Does long soak testing replace field observability?
It helps with time-dependent bugs, but it can't fully recreate real power, real environments, or real fleet scale. Soak testing and field observability are complementary — one stresses uptime on the bench, the other reads what actually happens where the device lives.

This is an embedded field note. If you're chasing a bug that only shows up at the customer, the longer story and how I work live on my profile. — Ritesh | ritzylab.com