BlinkThorLabsThe Day Our CI Pipeline Caught a Silent AUTOSAR Config Regression
A missed PTP correction... no wait, wrong incident. This one's about a CAN baud rate nobody meant to touch.
Incident log. All times local. Severity: would have been high, caught before it mattered.
09:14 — Commit a3f21c9 lands on a feature branch: "Cleanup: remove unused CAN channel from gateway ECU config." Routine housekeeping, flagged as low-risk by the author. Diff touches one ARXML file. Three lines changed. Nobody requests a careful review — it's a deletion, and deletions feel safe.
09:17 — PR opened. Two approvals come in within the hour, both from reviewers who, reasonably, read "remove unused channel" and check that the removed channel is, in fact, unused. It is. Nobody diffs the surrounding lines closely, because the surrounding lines weren't the stated intent of the change.
09:41 — Second approval lands. PR is mergeable. Author steps away for a coffee before hitting the button, which turns out to matter.
10:52 — Author returns, opens the PR to merge. Before clicking, habit (or luck) has them glance at the CI status tab one more time. It's not green. It's not red either — it's sitting in a state nobody on the team sees often: "config-regression-check: running (unusually long)."
10:58 — The regression check finishes. Red.
11:03 — First reaction, understandably: "But I only removed one channel, this has to be a flaky test." The PR gets re-queued to re-run CI, on the theory that something transient failed.
11:22 — Second run. Same failure, same specific stage: config-regression-check, comparing generated RTE output against a golden reference build. Not flaky. Consistent.
11:24 — Someone finally opens the actual diff report the pipeline generated, rather than the source diff in the PR. This is the moment the incident actually starts making sense, because the two diffs tell different stories.
11:31 — The regression report shows the generated Can_Cfg.c file changed in a way the PR's stated intent doesn't explain. Specifically: the default baud rate for the gateway's primary CAN controller shifted from 500 kbit/s to 250 kbit/s.
11:34 — Nobody remembers touching baud rate. A second look at the original three-line ARXML diff explains why: removing the "unused" channel also removed an override element that had been silently keeping the primary channel's baud rate at 500 kbit/s. Without that override, the container fell back to a different default defined further up the ARXML inheritance chain — one intended for a different, lower-speed variant of the ECU that shares the same base configuration template.
11:40 — This is the part worth sitting with: nothing in the three-line diff, read as a diff, looks wrong. You'd have to know that a specific override element was load-bearing for an unrelated parameter, and that removing it would cause an inherited default to silently take over. That relationship exists only in how the ARXML container hierarchy resolves values at generation time — it is not visible from reading the deleted lines in isolation.
11:47 — Someone finally says out loud what everyone's now thinking: if this had merged, the gateway ECU would have shipped configured for the wrong bus speed. Every other node on that CAN segment expects 500 kbit/s. A node quietly running at 250 would, in the field, either fail to communicate entirely or — worse, depending on transceiver behavior at mismatched rates — communicate just badly enough to cause an intermittent, hard-to-reproduce bus problem months later, the kind of bug that ends up needing a three-week investigation of its own.
12:15 — Root cause confirmed and written up. Fix identified: the "unused channel" removal needs to explicitly re-add the baud rate override at the level that actually matters, not rely on the container structure implicitly carrying it. Small fix. The kind of fix that's boring to write about, which is exactly the point.
13:02 — Fix committed, regression check re-run. Green. Generated Can_Cfg.c output now matches expectations exactly — same baud rate as before the "unused channel" was removed, channel itself confirmed gone.
13:20 — PR merged. Total time from first CI failure to safe merge: a little over two hours, almost entirely spent understanding why a three-line deletion produced an unrelated three-parameter change, rather than fixing anything once that was understood.
14:00 — Retro note added to the team's internal wiki, one sentence: "ARXML override removal can silently expose a different inherited default — diff the generated output, not just the source, for anything touching shared config containers."
Why This Needed a Machine to Catch It
The uncomfortable truth in this incident is that two competent engineers reviewed the actual source diff and correctly concluded it was safe, because it was safe as a diff — the three lines that were removed, read on their own, genuinely did only remove an unused channel. The baud rate regression wasn't a mistake in what was written. It was a consequence of AUTOSAR's own configuration inheritance model doing exactly what it's designed to do, applied to a container relationship that happened to be load-bearing in a way that wasn't obvious from the immediate context.
This is a structurally different category of risk from a typical code review miss. In application code, a reviewer can generally trace cause and effect by reading the function and its immediate callers. In an ARXML configuration with inheritance, defaults, and overrides spread across potentially many linked containers, the effective value of a parameter can depend on structure that's nowhere near the lines that were actually edited. Human review, done carefully and in good faith, is not well-suited to catching this class of regression — not because reviewers weren't paying attention, but because the information needed to catch it doesn't live in the diff they were looking at.
That's precisely the gap automated config-diff checking is supposed to close: not by reviewing intent, which humans are still better at, but by mechanically comparing the actual generated output — the thing that ends up on the ECU — against a known-good reference, every single time, regardless of how small or "obviously safe" the source change looks. The check doesn't need to understand that the change was meant to just remove a channel. It only needs to notice that the generated artifact changed somewhere it wasn't supposed to.
The Takeaway
Three lines. Zero code touched. A completely reasonable-sounding commit message. Two approvals from careful reviewers. And still, without an automated regression gate comparing generated RTE and BSW output against a golden reference, this would have shipped a wrong CAN bus speed to a gateway ECU — the kind of defect that doesn't announce itself in testing until it shows up as an intermittent field issue with no obvious cause months later.
If your AUTOSAR pipeline doesn't have a stage that diffs generated configuration output — not just source files — against a reference build on every PR, this is the incident that argument is worth having before you have your own version of this story to tell.
