A re-calibration method for object detection with multi-modal alignment bias in autonomous driving

Multi-modal object detection in autonomous driving has achieved great breakthroughs due to the usage of fusing complementary information from different sensors. The calibration in fusion between sensors such as LiDAR and camera was always supposed to be precise in previous work. However, in reality, calibration matrices are fixed when the vehicles leave the factory, but mechanical vibration, road bumps, and data lags may cause calibration bias. As there is relatively limited research on the impact of calibration on fusion detection performance, multi-sensor detection methods with flexible calibration dependency have remained a key objective. In this paper, we systematically evaluate the sensitivity of the SOTA EPNet++ detection framework and prove that even slight bias on calibration can reduce the performance seriously. To address this vulnerability, we propose a re-calibration model to re-calibrate the misalignment in detection tasks. This model integrates LiDAR point cloud, camera image, and initial calibration matrix as inputs, generating re-calibrated bias through semantic segmentation guidance and a tailored loss function design. The re-calibration model can operate with existing detection algorithms, enhancing both robustness against calibration bias and overall object detection performance. Our approach establishes a foundational methodology for maintaining reliability in multi-modal perception systems under real-world calibration uncertainties.