What this is. Opinion + Experience + Fact (25% opinion · 45% experience · 30% fact). Written in collaboration with AI — I discuss, I do not outsource.

The full architecture behind the short version on LinkedIn: how a firmware fix actually reaches a device in the field — and how it is kept from bricking the whole fleet.

Over-the-air update is the feature that lets you fix a product after it is already in someone's hands. It is also the only feature that can break every unit you ever shipped, at the same moment, from a single click. That asymmetry is why OTA deserves real respect: the upside is convenience, the downside is a field recall.

The trap is that "it uploaded fine" feels like success. It isn't. Delivery is the easy ten percent. The other ninety is knowing the new build landed on the right devices, installed cleanly, actually runs, and can be undone the instant it doesn't. Doing that safely is not one component. It is three, each with one job, and most teams underestimate at least one of them.

1. The problem: one push, the whole fleet

A bad firmware build in a lab is a bad afternoon. The same build pushed to a hundred thousand devices is a bad quarter — trucks, RMAs, and a support queue that never empties. And embedded devices are unforgiving: many have no screen, no keyboard, and no one on site. If a unit boots into a broken image and can't recover on its own, it is a brick, and someone has to physically touch it.

So the real design goal of OTA is not "deliver a file." It is: change the software on a remote device you cannot physically reach, and guarantee it can always get back to a working state. Everything below serves that one sentence.

2. The shape: three cooperating parts

A safe OTA system is three layers, and skipping any one is where fleets get bricked: an update server in the cloud decides and stages; an update agent on the device fetches and prepares; and a bootloader underneath does the actual switch and, crucially, the recovery.

The three layers of a safe OTA update A signed image flows left to right: from the update server in the cloud, to the update agent on the device, which writes it to the spare slot B while slot A keeps running; then the bootloader switches to B, boots it pending, and either confirms it healthy or reverts to A. A signed image, all the way to the metal UPDATE SERVER A · running B · writing… UPDATE AGENT A B BOOTLOADER confirm or revert

3. Layer one — the update server

The server is the brain of the campaign. It is the OTA engine inside your broader fleet-management platform, and it owns four things: an artifact store that holds the signed images with their hardware-compatibility metadata; a record of which device is on which version; a campaign service that decides who gets what and when; and the telemetry to know how it's going.

The single most important thing the server does is refuse to update the whole fleet at once. A staged (phased) rollout ships to a small group first — one percent, then ten, then the rest — and watches. If error rates or crash reports climb on the early cohort, the rollout halts automatically, and the blast radius was one percent, not one hundred. (This is the model behind Eclipse hawkBit, Mender's Artifact-and-deployment flow, and AWS IoT Device Management jobs, generalized.)

▸ First principle. Decide and stage in one place, or you are updating on a hope.

4. Layer two — the update agent

The agent is the small piece of software on the device that talks to the server. It polls or listens, and when an update is assigned it does four careful things, in order.

It pulls the image over whatever transport the device has — MQTT, HTTPS, CoAP — and the transfer is resumable, because field networks drop mid-download and you cannot afford to start from zero every time. It verifies the signature and the hash before trusting a single byte, so a corrupted or tampered image is rejected at the door. It writes the new image to the spare slot — never over the copy that is currently running. And it reports status back so the server's campaign view is real.

The discipline that matters most is the third one: the agent stages onto the inactive slot and leaves the live system untouched. Until the very last moment, nothing has been risked — if anything looks wrong, the device is still running the known-good build.

▸ First principle. Stage onto the spare; never overwrite what is keeping the device alive.

5. Layer three — the bootloader (where it gets device-specific)

Everything above is roughly the same on any device. The actual switch is not — it depends on the hardware, and this is the layer teams most often get wrong.

On a microcontroller, the pattern is A/B slots in flash and a small secure bootloader. With MCUboot, the bootloader swaps to the new image and boots it in a pending state; the application must call boot_set_confirmed() once it proves itself, or on the next reboot MCUboot reverts to the previous image. ESP-IDF does the same with two app partitions and an OTA-data partition: the new build must call esp_ota_mark_app_valid_cancel_rollback() within a timeout, and if it crashes or never confirms, the bootloader falls back to the last good slot. A watchdog underwrites all of it.

On embedded Linux, the same idea scales up to dual A/B root filesystems: the bootloader (often U-Boot, using a boot-count) picks the slot, and a framework — RAUC, Mender, or SWUpdate — manages the bundle and the atomic switch. RAUC is the lightest and most security-focused; SWUpdate the most flexible; Mender the most complete out of the box. Different tools, identical principle: two copies, an atomic pointer flip, automatic rollback.

And this is the layer where the standard meets the silicon. The idea — two slots, swap, roll back — is universal, but the implementation is vendor-specific: exactly how the image is copied into the spare slot, and how the swap is committed, differs from one chip family to the next. So does the security wrapped around it, and that security is anchored in hardware you cannot reproduce in software. A chain of trust starts inside the chip — immutable ROM boot code verifies the next stage's signature, which verifies the next, all the way up to your application — with the root key burned into one-time-programmable fuses or held in a secure element (NXP EdgeLock, Microchip ATECC608). Secure boot means only firmware signed with your key will run; flash encryption (for example ESP32's AES-256 flash encryption paired with Secure Boot v2) means no one can read your firmware off the chip or slip their own onto it. You inherit these from the vendor; you do not invent them. Which is the real lesson of this layer: "just switch the partition" quietly becomes a chip-by-chip engineering problem, and the trust anchor always lives in the silicon.

▸ First principle. Every update must be able to undo itself — without a human, and without a network.

6. Who owns rollback

Rollback is the most important word in OTA, and it is tempting to assume one component owns it. It doesn't. It lives in two places, at two scopes, and a safe system needs both.

The device rolls back itself. The bootloader keeps a boot-attempt counter and boots every new image on probation. If the new build fails to confirm it is healthy — it crashes, hangs, or never checks in — the bootloader reverts to the last good slot on its own, with no server and no network involved. It also enforces anti-rollback, refusing to boot an older, revoked image. This is local, automatic, and immediate: the single unit saves itself.

The server rolls back the fleet. A build can be individually "healthy" on every device and still be wrong — a subtle regression that only shows up in aggregate. That is the update server's job: watch the staged cohort, and if the telemetry turns, halt the campaign and roll the fleet back to the previous version. This is a decision made across thousands of devices, not on any one of them.

The device recovers the unit; the server recovers the population. Miss the first and a bad boot bricks a device in the field; miss the second and a bad build spreads across the fleet before anyone notices.

Rollback has two scopes Left, the device: the bootloader reverts an unhealthy slot B back to the good slot A by itself, with no network. Right, the fleet: the update server watches the cohort and, if telemetry turns, halts the campaign and rolls the whole fleet back to the previous version. The device saves itself bootloader · local Agood Bunhealthy auto-revert The server saves the fleet campaign · halt + roll back SERVER halt + roll the cohort back to the last good version

▸ First principle. Rollback has two scopes — the device saves itself, the server saves the fleet — and you need both.

7. The three rules that hold everywhere

Strip away the vendors and the chips and every safe OTA system obeys the same three rules:

  • Signed — the device verifies the image's signature before it will run it. An update path you don't authenticate is a remote-code-execution path you built on purpose.
  • Atomic — the switch either fully happens or fully doesn't. You never modify the image that is currently running; you prepare a second one and flip to it in a single, indivisible step.
  • Reversible — the new image boots on probation and must confirm it is healthy, or the system reverts on its own. Recovery is not a feature you add later; it is the first thing you design.

Add staged rollout on top — prove it on a few before it ever reaches all — and you have the whole discipline in four words.

▸ First principle. Signed, atomic, reversible, staged — miss one and OTA stops being a feature and becomes a liability.

8. Plumbing you shouldn't rebuild — so we did

Here is the uncomfortable part, and it is the same for every connected product: none of this is your product. The update server, the resumable-and-verified agent, the A/B bootloader with rollback — every connected device needs this exact stack, and it behaves the same whether the product is a thermostat or an industrial gateway. Yet most teams rebuild it from scratch, a little worse each time, in the months they meant to spend on the thing customers actually pay for.

So we built it once, properly — the whole three-layer stack, above the RTOS on the device and above the cloud — and open-sourced it. It is called EmbedIQ. You get the safe-update machinery as a foundation instead of a side quest, and your engineers spend their months on your product, not on rediscovering why you always need a spare slot. The architecture and the code are public — read them, and judge the quality for yourself.

▸ First principle. Demonstrated beats claimed — the plumbing you can inspect is the plumbing you can trust.

OTA is the clearest example of a rule that runs through everything we build: the impressive part is not shipping the update — it is guaranteeing the device can always come back. Get the three layers right, obey the three rules, and "push to fleet" stops being the scariest button in the building.

The short version of this is on my LinkedIn; the longer story and the links are on my profile.