What this is. Opinion + Experience + Fact (30% opinion · 35% experience · 35% fact). Written in collaboration with AI — I discuss, I do not outsource.

Why a device's own clock can't be trusted as the record of when things happened, and how to make time reliable across a fleet. The short version: cheap clocks drift and disagree, so you sync the wall clock to an outside source, use a monotonic clock for durations, and always know how stale your last sync is.

Unsynced clocks disagree; a sync gives a fleet one shared time Three device clocks show different times and drift apart. A periodic sync from an outside time source pulls them onto one shared wall-clock time, while each device still uses its own monotonic clock to measure durations. Three devices, three different "nows" — until they sync Left to themselves: each clock drifts device A device B device C After a sync: one shared wall-clock time time source (NTP / GPS / server on connect) "when" comes from the synced wall clock · "how long" comes from each device's monotonic clock

Here's a scene every fleet operator eventually lives through. Something goes wrong across a batch of devices, so you pull the logs from ten of them and try to line up what happened, in order. You can't. Every device's clock says something slightly different, and one or two are wildly off — one insists the event happened in 1970. The data is all there; it just can't be reconciled, because there's no shared, trustworthy sense of "when."

The lesson underneath it is simple and easy to forget: in a fleet, time is not a given. It's something you have to establish. Here's how.

1. Clocks drift, and they disagree

The clock inside a device is a counter driven by a small oscillator, and cheap oscillators drift — a little every hour, and more as temperature swings. A device without a battery-backed real-time clock is worse off: on every power cycle it boots with no idea what time it is, often starting from the Unix epoch (January 1, 1970). Put a thousand of these in the field and no two clocks agree. Now every timestamp you collect is measured against a different, moving ruler, and the question "what happened first?" has no reliable answer.

▸ First principle. An unsynced clock turns every timestamp into a guess.

2. Sync the wall clock; trust the monotonic one

There are really two kinds of time on a device, and they do different jobs. The wall clock is calendar time — "March 3rd, 14:22 UTC" — and it's what you sync to an outside source: an NTP server, a time signal from your backend on connect, or GPS. Because it drifts, you re-sync periodically rather than setting it once. The monotonic clock is a counter that only ever moves forward and never jumps; it's meaningless as a date, but it's the right tool for measuring how long something took, because it isn't affected by a sync correction yanking the wall clock backward or forward mid-measurement.

Mixing these up causes real bugs. Measure a timeout with the wall clock and a mid-measurement NTP correction can make a five-second wait look negative, or like an hour. The rule is boring and reliable: monotonic time for "how long," synced wall time for "when."

▸ First principle. Monotonic time for "how long," synced time for "when."

3. Know the uncertainty, and design for no-time

Even a synced device isn't perfectly on time — it's only as accurate as its last sync, drifting a little more with every hour that passes. So a device should track not just the time, but how stale that sync is, and pass that confidence along with its data. A timestamp that arrives with "last synced 40 minutes ago" is worth far more during an investigation than one that pretends to be exact.

And there's a moment every device passes through that quietly breaks systems: the window after boot, before the first sync, when it does not know the real time at all. Don't sign a message, expire a certificate, order events, or make a security decision on a clock you haven't set yet — because "1970" will sail straight past a naive expiry check. Decide explicitly what the device does before it trusts its clock. That on-device discipline about what a device knows and how sure it is of it is exactly the kind of thing EmbedIQ, my open framework for embedded and edge, is built around — because a trace is only as useful as its timestamp.

▸ First principle. A device should know not just the time, but how sure it is of it.

Sources

The mechanisms here — oscillator drift, devices booting at the Unix epoch without an RTC, NTP/GPS wall-clock synchronization, the distinction between monotonic and wall-clock time, and clock-uncertainty tracking — are standard timekeeping and distributed-systems practice. On why monotonic clocks exist and how they differ from wall time, see the NTP project documentation. This piece is method and lived experience; it contains no product performance figures.

FAQ

Why can't you trust a device's own clock?
Because the clock is driven by a small oscillator that drifts, and a device without a battery-backed real-time clock boots with no idea what time it is. Across a fleet, no two clocks agree, so timestamps measured against them can't be reliably compared or ordered.

What's the difference between a monotonic clock and a wall clock?
The wall clock is calendar time (a date and time) that you sync to an outside source and that can jump when corrected. The monotonic clock is a counter that only ever moves forward and never jumps; it's the right tool for measuring how long something took, while the wall clock answers when something happened.

How do embedded devices keep accurate time?
They sync the wall clock to an external source — an NTP server, a backend time signal on connect, or GPS — and re-sync periodically because the clock drifts. Durations are measured with the monotonic clock, and the device tracks how long ago the last sync was so it knows how much to trust its own time.

What should a device do before its clock is synced?
Treat its time as unknown. Don't sign messages, check certificate expiry, order events, or make security decisions on an unsynced clock, because an epoch time of 1970 will pass naive checks. Define explicitly what the device is allowed to do before its first successful sync.

This is an IoT field note. If your fleet's logs never quite line up after an incident, the longer story and how I work live on my profile. — Ritesh | ritzylab.com