# CalibFormer: A Transformer-based Automatic LiDAR-Camera Calibration Network

arXiv: https://arxiv.org/abs/2311.15241 · HTML: https://mlsanity.com/paper/2311.15241/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 26 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package](https://mlsanity.com/paper/2109.07982/) (2021)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [AutoAlignV2: Deformable Feature Aggregation for Dynamic Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2207.10316/) (2022)
- [LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2104.10831/) (2021)
- [Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups](https://mlsanity.com/paper/2101.04431/) (2021)
- [From One to Many: Dynamic Cross Attention Networks for LiDAR and Camera Fusion](https://mlsanity.com/paper/2209.12254/) (2022)

## Built on by (most cited first, 11 of 11)

- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Timealign: A multi-modal object detection method for time misalignment fusing in autonomous driving](https://mlsanity.com/paper/2412.10033/) (2024)
- [CalibRefine: Deep Learning-Based Online Automatic Targetless LiDAR-Camera Calibration with Iterative and Attention-Driven Post-Refinement](https://mlsanity.com/paper/2502.17648/) (2025)
- [BEVCALIB: LiDAR-Camera Calibration via Geometry-Guided Bird's-Eye View Representations](https://mlsanity.com/paper/2506.02587/) (2025)
- [MamV2XCalib: V2X-based Target-less Infrastructure Camera Calibration with State Space Model](https://mlsanity.com/paper/2507.23595/) (2025)
- [A re-calibration method for object detection with multi-modal alignment bias in autonomous driving](https://mlsanity.com/paper/2405.16848/) (2024)
- [UniCalib: Targetless LiDAR-Camera Calibration via Probabilistic Flow on Unified Depth Representations](https://mlsanity.com/paper/2504.01416/) (2025)
- [Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations](https://mlsanity.com/paper/2603.29414/) (2026)
- [RECO: Region-Aware Compensation for Extrinsic Perturbations in Roadside 3D Detection](https://mlsanity.com/paper/2607.20947/) (2026)
- [CalibBEV: LiDAR-Camera Calibration via BEV Alignment](https://mlsanity.com/paper/2608.02309/) (2026)
- [NCGR: Noise-Conditional Gated Rectification for Camera Extrinsic Perturbations in BEV 3D Object Detection](https://mlsanity.com/paper/2608.03895/) (2026)

## Abstract (arXiv)

The fusion of LiDARs and cameras has been increasingly adopted in autonomous driving for perception tasks. The performance of such fusion-based algorithms largely depends on the accuracy of sensor calibration, which is challenging due to the difficulty of identifying common features across different data modalities. Previously, many calibration methods involved specific targets and/or manual intervention, which has proven to be cumbersome and costly. Learning-based online calibration methods have been proposed, but their performance is barely satisfactory in most cases. These methods usually suffer from issues such as sparse feature maps, unreliable cross-modality association, inaccurate calibration parameter regression, etc. In this paper, to address these issues, we propose CalibFormer, an end-to-end network for automatic LiDAR-camera calibration. We aggregate multiple layers of camera and LiDAR image features to achieve high-resolution representations. A multi-head correlation module is utilized to identify correlations between features more accurately. Lastly, we employ transformer architectures to estimate accurate calibration parameters from the correlation information. Our method achieved a mean translation error of 0.8751 cm and a mean rotation error of 0.0562 circ on the KITTI dataset, surpassing existing state-of-the-art methods and demonstrating strong robustness, accuracy, and generalization capabilities.