Sensor fusion at scale: annotating for autonomy

Camera, radar and LiDAR each see the world differently. Keeping their annotations in sync across vendors is the unglamorous problem that decides whether a perception stack actually works.

Most write-ups about autonomous-vehicle annotation focus on the labeling task itself — draw the box, track the object. The harder problem, and the one that actually determines model quality, is keeping labels consistent across sensor modalities that disagree with each other by design.

Why cross-sensor drift happens

A camera frame and a LiDAR sweep captured at the "same" timestamp are rarely perfectly aligned — clock skew, rolling shutter, and differing capture rates all introduce small offsets. If your annotation pipeline treats each sensor independently, you end up with an object that's a pedestrian in the camera frame and an unclassified point cluster in the LiDAR frame three centimeters away. The model learns to distrust exactly the redundancy that sensor fusion is supposed to provide.

What a synchronized pipeline actually requires

We annotate in a shared 3D scene reconstruction rather than per-sensor, so every annotator is placing one object in one coordinate space, then checking that its projection lands correctly in every 2D camera view. Disagreements between projected and native-camera boxes get flagged automatically and routed to a calibration review — which, more often than we'd like, surfaces an actual extrinsic calibration drift on the vehicle itself, not an annotation error.

The long-tail problem this actually solves

The scenarios that break AV perception stacks — a cyclist partially occluded by a parked truck, a plastic bag that moves like debris — are exactly the cases where single-sensor annotation disagrees most with itself. Synchronized, cross-checked annotation is what turns those rare scenarios into usable training signal instead of noise the model has to average away.