# End-to-End Lidar-Camera Self-Calibration for Autonomous Vehicles

arXiv: https://arxiv.org/abs/2304.12412 · HTML: https://mlsanity.com/paper/2304.12412/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [A Survey of End-to-End Driving: Architectures and Training Methods](https://mlsanity.com/paper/2003.06404/) (2020)
- [Joint Camera Intrinsic and LiDAR-Camera Extrinsic Calibration](https://mlsanity.com/paper/2202.13708/) (2022)
- [Testing and verification of neural-network-based safety-critical control software: A systematic literature review](https://mlsanity.com/paper/1910.06715/) (2019)
- [CFNet: LiDAR-Camera Registration Using Calibration Flow Network](https://mlsanity.com/paper/2104.11907/) (2021)
- [SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera](https://mlsanity.com/paper/2207.03704/) (2022)

## Built on by (most cited first, 2 of 2)

- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [MamV2XCalib: V2X-based Target-less Infrastructure Camera Calibration with State Space Model](https://mlsanity.com/paper/2507.23595/) (2025)

## Abstract (arXiv)

Autonomous vehicles are equipped with a multi-modal sensor setup to enable the car to drive safely. The initial calibration of such perception sensors is a highly matured topic and is routinely done in an automated factory environment. However, an intriguing question arises on how to maintain the calibration quality throughout the vehicle's operating duration. Another challenge is to calibrate multiple sensors jointly to ensure no propagation of systemic errors. In this paper, we propose CaLiCa, an end-to-end deep self-calibration network which addresses the automatic calibration problem for pinhole camera and Lidar. We jointly predict the camera intrinsic parameters (focal length and distortion) as well as Lidar-Camera extrinsic parameters (rotation and translation), by regressing feature correlation between the camera image and the Lidar point cloud. The network is arranged in a Siamese-twin structure to constrain the network features learning to a mutually shared feature in both point cloud and camera (Lidar-camera constraint). Evaluation using KITTI datasets shows that we achieve 0.154 deg and 0.059 m accuracy with a reprojection error of 0.028 pixel with a single-pass inference. We also provide an ablative study of how our end-to-end learning architecture offers lower terminal loss (21% decrease in rotation loss) compared to isolated calibration