YOLOv8 on an Embedded GPU: Squeezing Real-Time Inference Out of 15 Watts

Model optimization matters more than your motion model. Here are the numbers.

Test objective: determine whether a YOLOv8-based object detector can hit real-time inference (defined here as ≥20 FPS sustained, end-to-end latency under 50ms) on automotive-grade embedded hardware within a 15W power envelope, without an unacceptable accuracy loss.

Test scope: YOLOv8n and YOLOv8s variants, three optimization stages (baseline FP32, INT8 quantization, INT8 + structured pruning), evaluated across three hardware targets. All numbers below are from our own benchmark runs on a held-out 4,000-image automotive validation set (mixed daytime/dusk urban and highway scenes), not vendor-published figures.

Baseline: Where We Started

YOLOv8s at native FP32 precision, no optimization applied, running on a workstation-class GPU for reference:

This is the accuracy ceiling we're measuring degradation against. The workstation number isn't a candidate deployment target — it's a reference point, since a discrete workstation GPU draws well over 100W and isn't relevant to an automotive power budget.

Hardware Candidates

Stage 1: INT8 Quantization

Post-training INT8 quantization applied to YOLOv8s using a calibration set of 500 representative automotive scenes (mixed lighting conditions, deliberately including some dusk/low-light frames since these are historically where quantization hurts accuracy most).

INT8 quantization alone more than doubled throughput on every platform, at a consistent accuracy cost in the 1.3-3.1 point mAP range — in line with what's typically reported for well-calibrated post-training quantization on detection models, and small enough that we considered it acceptable on its own merits before layering further optimization.

Worth flagging: the ASIC's larger accuracy drop (3.1 points versus Jetson's 1.4) is consistent with it having no fallback higher-precision path — quantization-sensitive layers that Jetson's mixed-precision engine can selectively keep at FP16 are forced to INT8 on the ASIC regardless, which shows up directly in the mAP delta.

Stage 2: TensorRT Conversion (Jetson only — platform-specific)

TensorRT conversion applies additional graph-level optimization — layer fusion, kernel auto-tuning, precision calibration refinement — on top of the INT8 quantization already applied. This step is Jetson-specific in our test since TensorRT is NVIDIA's own inference runtime; the SA8155P and ASIC use their own respective vendor toolchains (SNPE and a proprietary compiler, respectively), which we cover separately below.

TensorRT's gain here comes almost entirely from graph-level efficiency (layer fusion, memory layout optimization) rather than any further precision reduction, which explains the negligible additional accuracy cost (0.2 points) compared to the much larger jump from FP32 to INT8 in Stage 1.

Stage 3: Structured Pruning

Structured channel pruning applied to the INT8+TensorRT model, targeting a 30% reduction in channel count in the backbone's later stages (where we found the least accuracy sensitivity via a layer-wise sensitivity analysis pass before committing to the pruning ratio).

Pruning delivers a meaningfully smaller relative gain than quantization did (roughly 20-25% additional throughput versus quantization's 100%+ improvement), at a proportionally larger accuracy cost per unit of speed gained. This is the point in the optimization pipeline where the trade-off curve visibly steepens — the easy accuracy-for-speed exchanges have already been made in Stage 1.

Full Comparison Table

Reading the Results

Every platform clears the 20 FPS / 50ms real-time bar after Stage 1 alone. This is worth emphasizing because it means the "must we prune" question is not actually about hitting a minimum real-time bar — all three platforms do that with INT8 quantization alone. Pruning becomes relevant only if your target frame rate is meaningfully above the minimum (for redundancy margin, multi-camera scaling, or headroom for other concurrent workloads sharing the same compute budget), not as a requirement to reach basic real-time operation.

The accuracy cost is front-loaded, not evenly distributed. Going from FP32 to INT8 costs 1.3-3.1 mAP points and buys more than double the throughput. Going from there through TensorRT/SNPE optimization to pruning costs a comparable or larger additional accuracy hit (1.8-2.7 further points) for a much smaller proportional speed gain. If your accuracy budget is tight, INT8 quantization plus compiler-level graph optimization is close to the efficient frontier on its own — pruning should be treated as an optional further step reserved for cases with genuine headroom requirements, not a default final stage.

The custom ASIC's power efficiency comes with the least flexibility. At 9.7W it's the most power-efficient of the three at comparable throughput, but it also has the largest accuracy cost at every stage, precisely because it offers no precision fallback — every layer runs INT8 whether or not that layer is quantization-sensitive. The Jetson's mixed-precision engine, by contrast, can selectively preserve accuracy-critical layers at higher precision, which shows up consistently as a smaller mAP delta at every corresponding optimization stage.

The Trade-off, Stated Plainly

There is no configuration in this test matrix that achieves workstation-level accuracy (71.2% mAP) at automotive power budgets (under 15W) and real-time frame rates simultaneously — that combination doesn't exist yet at this model scale on this hardware class. What does exist is a genuinely usable operating point: INT8 quantization plus vendor-level graph optimization, landing in the 68-70% mAP range across all three platforms, comfortably clearing real-time throughput requirements, within power budget with margin to spare.

Whether to push further into pruning territory is a decision that should be driven by your specific frame-rate margin requirements, not treated as an automatic next step in an optimization checklist — the accuracy-per-FPS exchange rate gets measurably worse the further you push past INT8 quantization and compiler-level optimization alone.