Camera-Radar Time Sync: The 40ms Bug

A missed PTP correction cost four car-lengths of tracking error at speed.

Case file: intermittent fusion tracking offset, high-speed scenarios only. Status: closed. Root cause: 40 milliseconds.

This one is written up as a timeline because that's genuinely how it unfolded — a symptom that looked like a calibration problem, an investigation that eliminated calibration, and a root cause that turned out to be so small a number that it was almost insulting how much damage it did.

Day 1 — The Report

A test engineer flags a tracking anomaly from a highway test session: the fused object track for a lead vehicle is drawing noticeably ahead of or behind the vehicle's actual visual position in the overlay — sometimes by what looked like a full car length, sometimes more. It's inconsistent. It doesn't show up at low speed. Both the camera perception pipeline and the radar pipeline, checked independently against their own raw outputs, are reporting sane, individually correct detections. The problem only appears once you fuse them.

That last detail is the whole case, in retrospect. Nobody caught it yet.

Day 2 — Ruling Out the Obvious

Extrinsic calibration is the first suspect, because a fusion offset that only shows up under certain conditions smells like a calibration error that's being masked at low speed and exposed at high speed — a small angular misalignment between camera and radar mounting can produce exactly that kind of speed-dependent-looking symptom if you don't think too hard about it.

We re-ran the extrinsic calibration procedure against a static target board at multiple distances. Reprojection error came back within spec — comfortably under a pixel at all tested ranges. We checked radar-to-camera boresight alignment against the vehicle's known mounting geometry. Also within spec. Calibration was not the problem, and by end of day, we had eliminated it with enough confidence to stop looking there.

Day 3 — Looking at the Actual Numbers Instead of the Overlay

Instead of staring at the visual overlay, we pulled the raw per-detection timestamps for both sensor streams during one of the anomalous sessions and just... looked at them side by side.

Camera frames were timestamped at capture using the system's PTP-synchronized clock. Radar detections were timestamped the same way, via the radar unit's own PTP client. In theory, both should be referencing the same grandmaster clock, correctable to sub-millisecond agreement. In practice, when we diffed timestamps for detections we could independently confirm referred to the same physical event — a hard deceleration spike visible in both the camera's optical flow and the radar's Doppler signature — the radar timestamp for that event was consistently offset from the camera timestamp by very close to 40 milliseconds, and the offset was stable across the session, not random jitter.

That's not calibration. That's clock disagreement.

Day 4 — Finding Where the 40ms Actually Came From

We pulled the PTP sync logs for the radar unit specifically. The radar's PTP client was configured to apply periodic offset correction against the grandmaster, but a configuration change from an earlier integration cycle — made for an unrelated reason, to reduce network chatter on a bandwidth-constrained segment — had extended the correction interval on that specific node well beyond its default. Between corrections, the radar unit's local clock, running on its own crystal oscillator, was accumulating drift relative to the grandmaster. By the time the next correction fired, accumulated drift had grown to approximately 40ms before being yanked back to true time — meaning for a meaningful fraction of every correction cycle, the radar's timestamps were meaningfully wrong, and the fusion pipeline had no way to know that.

The camera, on a different network segment with its default (much shorter) correction interval untouched, stayed tightly synced the whole time. Nobody had checked whether that one earlier config change, made for a completely different reason, had this consequence — because on its own, it looked like a harmless network optimization, not a timing correctness change.

Day 5 — Why 40 Milliseconds Is a Much Bigger Number Than It Sounds

Here's the arithmetic that makes this bug matter, rather than being a rounding error.

At 100 km/h (27.8 m/s), a target vehicle moves:

27.8 m/s × 0.040 s = 1.11 meters in 40 milliseconds

That's the position error introduced on the radar side alone, relative to true time, at the moment of peak accumulated drift. The camera stream, correctly synced, reports the target's position at the actual current time. The fusion algorithm, receiving a radar detection timestamped 40ms in the past (as far as true time is concerned) but treating it as current, effectively fuses "where the target was 40ms ago, according to radar" with "where the target is right now, according to camera" — as if both measurements described the same instant, when they described two different instants roughly a meter apart in target displacement.

At a closing speed scenario — our lead vehicle braking hard while our own vehicle continued at speed — relative velocity between the two vehicles was higher than either vehicle's absolute speed, which is exactly why the visible error looked like several car lengths rather than one: the 1.11m radar-side error compounds with the relative velocity term used in the fusion's association and prediction logic, not the raw ego-speed term, and relative velocities in closing scenarios routinely exceed 150 km/h even when neither vehicle is exotic. At that relative speed, the same 40ms offset corresponds to over 1.6 meters of relative displacement error — close to the observed several-car-length discrepancy once you account for how the tracking filter's prediction step compounds a bad timestamp across a few cycles before the association logic has enough new evidence to correct it.

The bug was never present at low speed in any visible way, for the boring reason that low relative speed times 40ms is a displacement small enough to sit inside normal tracking noise. It was a high-speed-only bug not because the underlying clock error was speed-dependent — it wasn't — but because the consequence of a fixed time error scales directly with target velocity, which is a distinction worth internalizing: a timing bug's visibility in your test matrix depends entirely on whether your test matrix includes the speed range where its consequence becomes larger than your noise floor.

Day 6 — The Fix

Two changes, deployed together:

Reverted the PTP correction interval on the radar node back to its default short interval, eliminating the accumulated drift window entirely. Post-fix logs showed radar-to-camera timestamp agreement within 1-2ms across extended sessions, consistent with expected PTP performance on a properly configured network.

Added a timestamp sanity-check layer in the fusion pipeline itself, independent of the PTP fix, so that a future clock sync regression wouldn't go undetected until someone eyeballed a tracking overlay again. The check compares expected versus observed timestamp deltas between sensor streams reporting the same physical event (using cross-modal event correlation, similar to how we found the bug in the first place) and raises a diagnostic flag if disagreement exceeds a defined threshold — turning what had been a multi-day forensic investigation into something that would now surface as an immediate, specific alert.

We re-ran the original highway test route that had first surfaced the anomaly. Tracking held tight through braking events at speed, with no visible offset, across five subsequent sessions.

Closing Notes

The case is worth remembering less for the specific fix and more for how much investigative effort went toward the wrong hypothesis first, for entirely reasonable reasons — a speed-dependent visual symptom looks exactly like a calibration problem until you specifically check the one thing calibration procedures don't check: whether your sensors agree, in absolute terms, about when something happened, not just where.

Multi-sensor fusion systems are built on an assumption that's easy to forget precisely because it's usually true: that "now" means the same thing across every sensor stream. The moment that assumption breaks, even by an amount that sounds trivial on a spec sheet, every downstream calculation that depends on relative motion — which, in ADAS, is most of them — inherits the error, scaled by whatever velocity happens to be in play. Test your timing assumptions at the speeds where they'll actually be punished, not just the speeds where they're convenient to validate.