We Automated 80% of Our HIL Regression Suite. Here's What Broke First.

A practical retrospective on what went well, what didn't, and what we only found because we went looking.

Six months ago we set out to automate our Hardware-in-the-Loop regression suite for our ADAS domain controller — at the time, a suite that took a rotating pair of engineers most of a working week to run manually, scenario by scenario, recording pass/fail by hand into a shared spreadsheet that everyone quietly hated. The goal was straightforward: automate as much of it as we reasonably could, run it unattended overnight and on every significant commit, and free up the engineers who'd been manually babysitting a test rig instead of doing engineering work.

We hit roughly 80% automation coverage, and it's been running in some form for four months now. This is the honest retrospective — not the version we'd put in a conference talk, the version we'd actually tell a team about to start the same project.

What Went Well

The headline number is the one everyone expects and the one that's genuinely true: our full regression pass went from roughly 34 hours of engineer-supervised manual execution spread across a work week, down to 6.5 hours of unattended automated execution, typically run overnight so results are waiting the next morning rather than consuming anyone's actual working day. That's not a modest improvement — it changed the rhythm of the whole team's relationship with regression testing, from "something we schedule and dread" to "something that just happens."

The less obvious but arguably more valuable win: defects surface earlier in the development cycle than they used to. With manual execution, the regression suite ran roughly once per sprint, because running it more often wasn't practical given the engineer-hours it consumed. With automation, it runs on every merge to our integration branch. We've caught three regressions in the last four months on the same day they were introduced, rather than up to two weeks later during the next scheduled manual pass — each of those would previously have required someone to bisect across a sprint's worth of commits to find the actual cause, instead of the automated run pointing directly at the commit that broke it.

We also, somewhat unexpectedly, ended up with better test documentation as a side effect of automation, not as a separate initiative. Writing a test case as an executable script forces a level of precision about exact inputs, exact expected outputs, and exact pass/fail criteria that a human-executed test procedure document often gets away with leaving slightly vague, because a human running the test can use judgment to fill the gaps. That precision turned out to be useful even independent of automation — a few ambiguities in our test specifications that had existed for years, quietly resolved by different engineers different ways each time they ran the test manually, got surfaced and had to be explicitly settled the moment we tried to encode them as scripted assertions.

What Didn't Go as Expected

The honest part: a meaningful fraction of our automated test cases turned out to be flaky, and the root cause was almost always some form of timing race in the automation framework itself, not in the system under test. Our HIL rig involves coordinating a real-time simulation platform, the ECU under test, and our automation harness across several communication channels, and getting the timing of "wait for the simulation to reach state X before sending stimulus Y" reliably right, across every scenario, took considerably more iteration than we budgeted for. At one point roughly 12% of our automated test cases had a nonzero, unpredictable failure rate that had nothing to do with the ADAS software being tested — purely automation-framework timing races — which meant engineers were spending real time re-investigating "failures" that turned out to be false alarms, undermining trust in the suite exactly during the period we most needed people to trust it.

Script maintenance cost also came in meaningfully higher than we scoped for. Every time the ADAS software's diagnostic interface, signal naming, or timing characteristics changed even slightly — which happens routinely as a codebase evolves — some subset of automated test scripts needed corresponding updates. We'd budgeted maintenance time based on a rough guess extrapolated from our manual test procedure update history, and the actual ongoing cost of keeping scripts in sync with a moving target has run noticeably above that estimate, closer to what we'd now describe as "a genuine, standing engineering responsibility" rather than the "occasional light touch-up" we'd initially planned staffing around.

Neither of these is a reason we'd undo the project. But if we were scoping this again, we'd budget meaningfully more time for framework timing robustness before declaring victory, and we'd staff ongoing script maintenance as an explicit, named responsibility from day one rather than something absorbed informally into whoever happened to notice a script had broken.

What Surprised Us

This is the finding that actually changed how we think about test automation generally, and it wasn't something we were looking for.

Manual test execution, as tedious as it was, had an accidental property we hadn't appreciated until it was gone: the engineer running the tests would naturally vary the order, timing, and minor procedural details from run to run — sometimes running scenario 14 before scenario 9, sometimes pausing slightly longer between steps because they got interrupted, sometimes executing a diagnostic check in a slightly different sequence out of personal habit. None of this was intentional test design. It was just normal human variability.

Our automation, once built, executed every scenario in exactly the same order, with exactly the same timing, every single run — which is usually treated as a feature (reproducibility) rather than a risk. It took a specific incident to reveal the risk side of that same property: a state-persistence bug, where a specific diagnostic session left a small piece of ECU state improperly reset, only manifested when Scenario A ran immediately before Scenario B in a specific order with a specific timing gap between them. Our automated suite always ran scenarios in the same fixed order — a different order than the specific sequence that triggered the bug — and had been silently passing through four months of nightly runs without ever encountering the exact ordering that would expose it. We only found it because a manual exploratory test session, done for an unrelated reason, happened to execute scenarios in a different sequence than our automated suite's fixed order, and hit the bug within an hour.

This was genuinely uncomfortable to sit with. Our automated suite hadn't just failed to catch this bug — its very consistency, the property we'd been treating as an unambiguous virtue, had been actively suppressing our ability to catch an entire class of ordering-dependent defect, because "run everything the same way every time" and "explore different orderings to find state-dependent bugs" turned out to be in direct tension, and we'd only optimized for the first one.

What We Changed as a Result

We didn't roll back automation — the throughput and early-detection wins are real and worth keeping. But we made three specific changes directly because of what we found:

We added a randomized scenario ordering mode, run on a rotating basis (not every night, but regularly and unpredictably) specifically to reintroduce the ordering variability that manual testing used to give us for free, rather than always executing our fixed, "efficient" sequence.

We now explicitly budget engineer time for periodic manual exploratory testing alongside the automated suite, rather than treating manual testing as something automation was meant to fully eliminate — treating it instead as covering a genuinely different risk category (unscripted, human-driven exploration) that scripted automation, by its nature, structurally cannot cover no matter how comprehensive the script coverage becomes.

And we now treat flaky test triage and script maintenance as a named, staffed, ongoing responsibility with dedicated time allocated in sprint planning, rather than the ad-hoc "whoever notices" arrangement we started with, which was the direct cause of most of our "what didn't go as expected" section above.

The Honest Summary

Eighty percent automation coverage was worth doing, and the throughput and early-detection benefits are genuinely large, not marketing-large. But this project's biggest lesson wasn't about automation framework engineering — it was a reminder that the informal, unstructured parts of a manual process sometimes carry real value that isn't obvious until you've removed them, and that a test suite's greatest strength (consistency, reproducibility) can simultaneously be a specific, structural blind spot for an entire category of defect. We'd do this project again. We'd just go into it knowing that "automate the test suite" and "replace all the value the old process provided" are not automatically the same project, even when they look like they should be.