Simulating Rain: Why Our LiDAR Model Lied to Us in CARLA

Six weeks of validation numbers told us we'd solved rain. They were answering a different question than the one we asked.

For about six weeks, we believed we had solved rain. Not "solved" in the marketing sense — solved in the specific, measurable sense that matters to a perception team: our object detection model, trained and validated extensively in CARLA's simulated rain scenarios, held its mAP within two points of clear-weather performance across every synthetic rain intensity we threw at it. Light drizzle, moderate rain, heavy downpour — the numbers barely moved. We had graphs. We had a slide deck. We were, briefly, quite pleased with ourselves.

What We Thought We Knew

The validation pipeline looked, by every internal metric we tracked, genuinely solid. We ran our full detection stack against CARLA's built-in weather presets across several thousand synthetic frames, comparing LiDAR-based detection performance in rain against the same scenes rendered clear. The model's degradation curve was gentle — a small, expected dip in detection confidence at longer range, nothing that looked alarming, nothing that looked like the kind of cliff-edge failure that shows up in perception postmortems.

We took this as validation that our sensor fusion pipeline — which leaned on LiDAR fairly heavily specifically because camera performance is well known to suffer more in rain — was doing its job. The whole design philosophy, in fact, was built around LiDAR being the more rain-robust modality precisely because it doesn't depend on visible-light contrast the way a camera does. The simulation results seemed to confirm the philosophy was sound.

The First Loose Thread

The first hint that something was off didn't come from a rain test at all. It came from a routine review of CARLA's sensor model documentation, done by an engineer who was actually trying to solve an unrelated problem — tuning point cloud density for a different scenario — and who happened to read, closely, exactly how CARLA's LiDAR sensor simulates atmospheric interaction.

The note, once you actually go looking for it, is not hidden. CARLA's LiDAR rain/fog attenuation model applies a relatively simplified per-point dropout and intensity reduction based on precipitation density, applied fairly uniformly across the scene. It does not model multi-path reflection — the phenomenon where a laser pulse bounces off a raindrop or a wet road surface at an angle and returns a spurious, geometrically misleading point rather than simply attenuating or vanishing. It also doesn't model distance-dependent, non-uniform attenuation with the fidelity that real atmospheric scattering physics actually exhibits, where signal loss compounds nonlinearly with range in a way that a flat dropout-probability model doesn't fully capture, particularly at the longer detection ranges that matter most for highway-speed obstacle detection.

At the time, this note landed as an interesting caveat. Nobody on the team, including the person who found it, immediately connected it to our six weeks of reassuring validation numbers. That connection took another two weeks, and it took a field test to make it undeniable.

The Twist

Here's the part that reframes everything above: our model's gentle, reassuring degradation curve in CARLA rain wasn't evidence that our perception stack handled rain well. It was evidence that CARLA's rain wasn't actually testing the thing we thought it was testing.

The model had, over the course of training and validation, been implicitly optimized against a specific, simplified statistical signature of "what LiDAR data looks like in rain" — a signature defined by CARLA's dropout-and-attenuation model, not by the real, messier physics of multi-path reflection and nonlinear range-dependent scattering. In effect, we hadn't trained a model that was robust to rain. We had trained a model that was mildly robust to a specific synthetic noise distribution that happened to be labeled "rain" in our simulation config, and which turned out to be measurably easier to handle than the real thing — smoother, more uniform, more predictable, and missing an entire category of spurious returns that real wet-weather LiDAR data is full of.

The field test that surfaced this ran during an actual rainstorm on our test track — not by original design, honestly; it was scheduled test time and the weather didn't cooperate with rescheduling. Detection performance didn't degrade gently the way our CARLA validation had promised. It dropped sharply, with a specific and telling failure signature: a meaningful increase in false-positive detections at mid-range, roughly consistent with spurious returns from multi-path reflections off the wet road surface — exactly the phenomenon our simulation had never modeled, appearing for the first time in data the model had never been prepared for.

The gentle degradation curve from six weeks of simulation had been, in retrospect, real evidence — just evidence about CARLA's rain model, not about our perception stack.

Why This Is an Easy Trap, Not a Careless One

It's worth being fair to how this happened, because the failure mode here isn't "the team was sloppy." It's a structural risk in any simulation-validated perception pipeline: a simulator's noise model becomes an implicit, invisible part of your model's training distribution, and if that noise model diverges from reality in a way that happens to make the simulated version of a hard condition easier than the real one, your validation metrics will confidently tell you a false story. The failure isn't visible in the validation numbers themselves — a smooth, mild degradation curve looks exactly like what "handling rain well" should look like. It only becomes visible in contrast with real-world data, which is precisely the data a simulation-heavy validation pipeline is trying to reduce reliance on.

The genuinely uncomfortable part: this kind of gap can exist in any simulated sensor model, for any weather or edge-case condition, and validation metrics computed entirely within the simulation will never surface it, because the metric and the simulation share the same blind spot by construction.

What We Changed

We didn't abandon CARLA — synthetic data remains genuinely valuable for scenario coverage and edge cases that are dangerous or impractical to collect in the real world. But we changed how we validate anything simulation-heavy:

Real-world data now has a mandatory, non-negotiable seat at every weather-condition validation gate, even a small amount — a rule that any claimed robustness to a specific condition (rain, fog, snow) needs at minimum a modest real-world validation set in that condition before it's considered validated, not just a large synthetic one.

We now specifically probe for known-divergent physics in any simulator's sensor model before trusting validation results built on it — reading the sensor model's actual implementation notes, not just its marketing description, has become a standing step before designing a validation plan around a new simulated condition.

Our metrics dashboard now flags — rather than just reports — validation runs performed entirely in simulation for safety-relevant conditions, specifically to stop a smooth, reassuring simulated curve from being treated with the same confidence as a real-world validated one, even unintentionally, in a slide deck six weeks from now.

Simulation didn't lie to us out of malice, and it didn't even lie about something we couldn't have checked. It answered a question we didn't realize we'd accidentally changed — and the six weeks in between finding that out cost us more than the actual fix, once we found it, ever did.