# RegNet: Multimodal Sensor Registration Using Deep Neural Networks

arXiv: https://arxiv.org/abs/1707.03167 · HTML: https://mlsanity.com/paper/1707.03167/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2017.

## Built on by (most cited first, 30 of 69)

- [EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba](https://mlsanity.com/paper/2403.09977/) (2024)
- [Multi-Modal 3D Object Detection in Autonomous Driving: a Survey](https://mlsanity.com/paper/2106.12735/) (2021)
- [Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups](https://mlsanity.com/paper/2101.04431/) (2021)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [CRLF: Automatic Calibration and Refinement based on Line Feature for LiDAR and Camera in Road Scenes](https://mlsanity.com/paper/2103.04558/) (2021)
- [MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal calibration](https://mlsanity.com/paper/2303.03056/) (2023)
- [OpenCalib: A Multi-sensor Calibration Toolbox for Autonomous Driving](https://mlsanity.com/paper/2205.14087/) (2022)
- [CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibration](https://mlsanity.com/paper/2402.00129/) (2024)
- [CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks](https://mlsanity.com/paper/2103.14793/) (2021)
- [PnLCalib: Sports Field Registration via Points and Lines Optimization](https://mlsanity.com/paper/2404.08401/) (2024)
- [SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields](https://mlsanity.com/paper/2311.15803/) (2023)
- [Online,Target-Free LiDAR-Camera Extrinsic Calibration via Cross-Modal Mask Matching](https://mlsanity.com/paper/2404.18083/) (2024)
- [Calib-Anything: Zero-training LiDAR-Camera Extrinsic Calibration Method Using Segment Anything](https://mlsanity.com/paper/2306.02656/) (2023)
- [CalibFormer: A Transformer-based Automatic LiDAR-Camera Calibration Network](https://mlsanity.com/paper/2311.15241/) (2023)
- [INF: Implicit Neural Fusion for LiDAR and Camera](https://mlsanity.com/paper/2308.14414/) (2023)
- [End-to-end 2D-3D Registration between Image and LiDAR Point Cloud for Vehicle Localization](https://mlsanity.com/paper/2306.11346/) (2023)
- [UniCal: Unified Neural Sensor Calibration](https://mlsanity.com/paper/2409.18953/) (2024)
- [SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator](https://mlsanity.com/paper/2109.10270/) (2021)
- [MULi-Ev: Maintaining Unperturbed LiDAR-Event Calibration](https://mlsanity.com/paper/2405.18021/) (2024)
- [Efficient Multimodal Dataset Distillation via Generative Models](https://mlsanity.com/paper/2509.15472/) (2025)
- [SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera](https://mlsanity.com/paper/2207.03704/) (2022)
- [From Chaos to Calibration: A Geometric Mutual Information Approach to Target-Free Camera LiDAR Extrinsic Calibration](https://mlsanity.com/paper/2311.01905/) (2023)
- [ShapeFormer: Shape Prior Visible-to-Amodal Transformer-based Amodal Instance Segmentation](https://mlsanity.com/paper/2403.11376/) (2024)
- [Uncertainty-Aware Online Extrinsic Calibration: A Conformal Prediction Approach](https://mlsanity.com/paper/2501.06878/) (2025)
- [UniCal: a Single-Branch Transformer-Based Model for Camera-to-LiDAR Calibration and Validation](https://mlsanity.com/paper/2304.09715/) (2023)
- [Robust Long-Range Perception Against Sensor Misalignment in Autonomous Vehicles](https://mlsanity.com/paper/2408.11196/) (2024)
- [Amodal Instance Segmentation with Diffusion Shape Prior Estimation](https://mlsanity.com/paper/2409.18256/) (2024)
- [Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning](https://mlsanity.com/paper/2206.02856/) (2022)
- [Contextual Active Model Selection](https://mlsanity.com/paper/2207.06030/) (2022)
- [Enhanced Low-resolution LiDAR-Camera Calibration Via Depth Interpolation and Supervised Contrastive Learning](https://mlsanity.com/paper/2211.03932/) (2022)

## Abstract (arXiv)

In this paper, we present RegNet, the first deep convolutional neural network (CNN) to infer a 6 degrees of freedom (DOF) extrinsic calibration between multimodal sensors, exemplified using a scanning LiDAR and a monocular camera. Compared to existing approaches, RegNet casts all three conventional calibration steps (feature extraction, feature matching and global regression) into a single real-time capable CNN. Our method does not require any human interaction and bridges the gap between classical offline and target-less online calibration approaches as it provides both a stable initial estimation as well as a continuous online correction of the extrinsic parameters. During training we randomly decalibrate our system in order to train RegNet to infer the correspondence between projected depth measurements and RGB image and finally regress the extrinsic calibration. Additionally, with an iterative execution of multiple CNNs, that are trained on different magnitudes of decalibration, our approach compares favorably to state-of-the-art methods in terms of a mean calibration error of 0.28 degrees for the rotational and 6 cm for the translation components even for large decalibrations up to 1.5 m and 20 degrees.