What this is. Opinion + Experience + Fact (35% opinion · 35% experience · 30% fact). Written in collaboration with AI — I discuss, I do not outsource.

Why a device that isn't complaining still isn't a device you can trust — and why field observability has to be built in, not bolted on. The short version: in the field you only ever know what the device recorded. If observability isn't on by default, your first real debugging session starts with no evidence.

A silent device could be healthy or dead; only telemetry tells you A quiet device sends nothing, so silence looks identical whether it is fine or has failed. A device with observability sends a steady heartbeat of telemetry, so you can tell the difference. Silence tells you nothing. A heartbeat tells you everything. Quiet device healthy? or dead? — you can't tell Observed device (EmbedIQ) a steady heartbeat you can read

On the bench, you watch the device the entire time. You have a debugger attached, a terminal open, eyes on the behaviour. In the field, all of that goes away. The device is in a customer's hands, or on a pole, or inside a machine, and you get exactly what it chose to tell you — which, for most devices, is nothing until something breaks badly enough to notice. And here's the trap that catches good teams: a device that isn't complaining feels like a device that's fine. It usually isn't. It's just quiet.

This is the observability problem in embedded, and it's the one I care about most, because it's the difference between a field failure you can explain in an afternoon and one you chase for weeks. Let me lay out why quiet is dangerous, why the recording matters more than the failure, and why observability has to be a default rather than a decision.

1. Absence of complaints is not absence of problems

A device can degrade in ways it will never announce. It drops one reading in fifty and averages over the gap. It quietly retries a connection all night and burns through a battery budget. It runs a little hotter, resets a little more often, answers a little slower. None of these throw an error; none of them page anyone. From the outside the device looks healthy, because "healthy" is being inferred from silence — and silence is not a measurement.

You find out eventually, of course. You find out from a support ticket, a return, a customer who's annoyed, a fleet number that finally crossed a line. By then the moment you actually needed to see — the conditions, the sequence, the state right before it went wrong — is long gone. (It's the same reason the bench and the field disagree: the field removes your ability to watch.)

▸ First principle. Silence from a device is not good news — it's data you chose not to collect.

2. You can only debug what the device already recorded

When a field failure finally lands on your desk, your entire investigation is bounded by one thing: what the device captured at the moment it happened. Not what you wish it had captured — what was actually being recorded when it broke. If the relevant signal wasn't being kept, it doesn't exist, and no amount of cleverness recovers it.

The usual response — "let's add some logging and wait for it to happen again" — is more expensive than it sounds. It means you pay for the failure twice: once when it hits a customer, and again while your schedule waits for a reproduction that may take weeks and may never come on demand. The debugging session you want is the one where the evidence was already there before anything went wrong. (Knowing which few numbers tell you a fleet is healthy is the same instinct, one level up.)

▸ First principle. In the field you don't debug the failure — you debug the recording of it.

3. Make observability the default, not an afterthought

If observability is something an engineer has to remember to add, module by module, under deadline, then it won't be there in the one place you didn't think to add it — which is exactly where the bug will be. The fix is to make it structural: build the device so it can always tell you what it did, without hand-wiring instrumentation into every corner, and stamp every trace with the device id and the firmware version, so the question "which build produced this?" is never a guess.

This is the core of what I've been building in the open as EmbedIQ, the Apache-2.0 framework for embedded and edge: a zero-instrumentation way to see what a device actually did at runtime, so the record is already there when you need it — and every trace carries the device id and firmware version that turn a vague field report into a specific one. That's the shipped, on-device piece today. Getting that record off the device and into a fleet-wide view is the direction it's heading, not a box I'm claiming is already ticked — and I'd rather tell you where the line is than blur it.

▸ First principle. Observability you have to remember to add is observability you won't have when it counts.

Sources

The mechanisms here — silent degradation, the field-vs-bench evidence gap, and the value of always-on tracing stamped with build identity — are standard embedded and site-reliability practice (observability as a property of the system, not an add-on). This piece is method and lived experience; it contains no product performance figures, and the EmbedIQ capabilities described are limited to what ships today (on-device, zero-instrumentation tracing with device-id and firmware stamping); fleet-wide aggregation is described explicitly as a direction, not a shipped feature.

FAQ

Why isn't a device that reports no errors necessarily healthy?
Because many failures are silent: dropped readings, background retries, extra resets, faster battery drain, slow thermal creep. None of these throw an error, so "healthy" ends up inferred from silence — and silence isn't a measurement. The device can be degrading the whole time and reporting nothing.

What is field observability for embedded devices?
It's the ability to know what a deployed device actually did at runtime — what it sensed, decided, and did — from records the device itself kept, rather than from a debugger you can't attach in the field. Good field observability is always on and captures the context before a failure, not just the error after it.

Why build observability in from the start instead of adding logging later?
Because in the field you can only debug what was already being recorded. Adding logging after a failure means waiting for it to happen again — on a customer and on your schedule. If observability depends on someone remembering to add it, it won't be present in the one place the bug actually is.

What does EmbedIQ provide for observability today?
EmbedIQ (open, Apache-2.0) provides zero-instrumentation, on-device observability — a way to see what a device did at runtime without hand-wiring instrumentation into every module — with each trace stamped by device id and firmware version. Aggregating that into a fleet-wide view is a direction it's heading, not a shipped feature today.

This is an embedded field note. If you're deciding how much observability to build into a device before it ships, the longer story and how I work live on my profile. — Ritesh | ritzylab.com