# What Really Matters for Learning-based LiDAR-Camera Calibration

arXiv: https://arxiv.org/abs/2501.16969 · HTML: https://mlsanity.com/paper/2501.16969/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 28 Jan 2025.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase](https://mlsanity.com/paper/2309.05573/) (2023)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [CMRNet: Camera to LiDAR-Map Registration](https://mlsanity.com/paper/1906.10109/) (2019)
- [FAST-LIVO: Fast and Tightly-coupled Sparse-Direct LiDAR-Inertial-Visual Odometry](https://mlsanity.com/paper/2203.00893/) (2022)
- [CRLF: Automatic Calibration and Refinement based on Line Feature for LiDAR and Camera in Road Scenes](https://mlsanity.com/paper/2103.04558/) (2021)
- [LiDAR-Camera Calibration using 3D-3D Point correspondences](https://mlsanity.com/paper/1705.09785/) (2017)
- [LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry](https://mlsanity.com/paper/1804.05178/) (2018)
- [Joint Camera Intrinsic and LiDAR-Camera Extrinsic Calibration](https://mlsanity.com/paper/2202.13708/) (2022)
- [DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow](https://mlsanity.com/paper/2203.09385/) (2022)
- [CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration](https://mlsanity.com/paper/2402.00129/) (2024)
- [CMRNet++: Map and Camera Agnostic Monocular Visual Localization in LiDAR Maps](https://mlsanity.com/paper/2004.13795/) (2020)
- [CamVox: A Low-cost and Accurate Lidar-assisted Visual SLAM System](https://mlsanity.com/paper/2011.11357/) (2020)
- [CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks](https://mlsanity.com/paper/2103.14793/) (2021)
- [CFNet: LiDAR-Camera Registration Using Calibration Flow Network](https://mlsanity.com/paper/2104.11907/) (2021)
- [Road is Enough! Extrinsic Calibration of Non-overlapping Stereo Camera and LiDAR using Road Information](https://mlsanity.com/paper/1902.10586/) (2019)
- [From Chaos to Calibration: A Geometric Mutual Information Approach to Target-Free Camera LiDAR Extrinsic Calibration](https://mlsanity.com/paper/2311.01905/) (2023)
- [Automatic Target-Less Camera-LiDAR Calibration From Motion and Deep Point Correspondences](https://mlsanity.com/paper/2404.17298/) (2024)
- [Rail Vehicle Localization and Mapping with LiDAR-Vision-Inertial-GNSS Fusion](https://mlsanity.com/paper/2112.08563/) (2021)
- [Robust Extrinsic Self-Calibration of Camera and Solid State LiDAR](https://mlsanity.com/paper/2302.06276/) (2023)

## Built on by (most cited first, 2 of 2)

- [CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning](https://mlsanity.com/paper/2603.15767/) (2026)
- [Joint Multi-Camera LiDAR Extrinsic Calibration via Learned Pairwise Initialization and Geometric Refinement](https://mlsanity.com/paper/2605.31576/) (2026)

## Abstract (arXiv)

Calibration is an essential prerequisite for the accurate data fusion of LiDAR and camera sensors. Traditional calibration techniques often require specific targets or suitable scenes to obtain reliable 2D-3D correspondences. To tackle the challenge of target-less and online calibration, deep neural networks have been introduced to solve the problem in a data-driven manner. While previous learning-based methods have achieved impressive performance on specific datasets, they still struggle in complex real-world scenarios. Most existing works focus on improving calibration accuracy but overlook the underlying mechanisms. In this paper, we revisit the development of learning-based LiDAR-Camera calibration and encourage the community to pay more attention to the underlying principles to advance practical applications. We systematically analyze the paradigm of mainstream learning-based methods, and identify the critical limitations of regression-based methods with the widely used data generation pipeline. Our findings reveal that most learning-based methods inadvertently operate as retrieval networks, focusing more on single-modality distributions rather than cross-modality correspondences. We also investigate how the input data format and preprocessing operations impact network performance and summarize the regression clues to inform further improvements.