# SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator

arXiv: https://arxiv.org/abs/2109.10270 · HTML: https://mlsanity.com/paper/2109.10270/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving](https://mlsanity.com/paper/2003.03653/) (2020)
- [Gated-SCNN: Gated Shape CNNs for Semantic Segmentation](https://mlsanity.com/paper/1907.05740/) (2019)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer](https://mlsanity.com/paper/1511.03240/) (2015)
- [CCMI : Classifier based Conditional Mutual Information Estimation](https://mlsanity.com/paper/1906.01824/) (2019)
- [CRLF: Automatic Calibration and Refinement based on Line Feature for LiDAR and Camera in Road Scenes](https://mlsanity.com/paper/2103.04558/) (2021)
- [SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera](https://mlsanity.com/paper/2003.04260/) (2020)
- [CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks](https://mlsanity.com/paper/2103.14793/) (2021)
- [C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation](https://mlsanity.com/paper/2005.08226/) (2020)
- [DRMIME: Differentiable Mutual Information and Matrix Exponential for Multi-Resolution Image Registration](https://mlsanity.com/paper/2001.09865/) (2020)
- [Extrinsic Calibration of a 3D-LIDAR and a Camera](https://mlsanity.com/paper/2003.01213/) (2020)
- [LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network](https://mlsanity.com/paper/2012.13901/) (2020)
- [Experimental Evaluation of 3D-LIDAR Camera Extrinsic Calibration](https://mlsanity.com/paper/2007.01959/) (2020)

## Built on by (most cited first, 7 of 7)

- [3DGS-ReLoc: 3D Gaussian Splatting for Map Representation and Visual ReLocalization](https://mlsanity.com/paper/2403.11367/) (2024)
- [UniCal: Unified Neural Sensor Calibration](https://mlsanity.com/paper/2409.18953/) (2024)
- [Contrastive Learning of Features between Images and LiDAR](https://mlsanity.com/paper/2206.12071/) (2022)
- [Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning](https://mlsanity.com/paper/2206.02856/) (2022)
- [Enhanced Low-resolution LiDAR-Camera Calibration Via Depth Interpolation and Supervised Contrastive Learning](https://mlsanity.com/paper/2211.03932/) (2022)
- [CalibRefine: Deep Learning-Based Online Automatic Targetless LiDAR-Camera Calibration with Iterative and Attention-Driven Post-Refinement](https://mlsanity.com/paper/2502.17648/) (2025)
- [Improving Extrinsics between RADAR and LIDAR using Learning](https://mlsanity.com/paper/2305.10594/) (2023)

## Abstract (arXiv)

This paper proposes SemCal: an automatic, targetless, extrinsic calibration algorithm for a LiDAR and camera system using semantic information. We leverage a neural information estimator to estimate the mutual information (MI) of semantic information extracted from each sensor measurement, facilitating semantic-level data association. By using a matrix exponential formulation of the se(3) transformation and a kernel-based sampling method to sample from camera measurement based on LiDAR projected points, we can formulate the LiDAR-Camera calibration problem as a novel differentiable objective function that supports gradient-based optimization methods. We also introduce a semantic-based initial calibration method using 2D MI-based image registration and Perspective-n-Point (PnP) solver. To evaluate performance, we demonstrate the robustness of our method and quantitatively analyze the accuracy using a synthetic dataset. We also evaluate our algorithm qualitatively on an urban dataset (KITTI360) and an off-road dataset (RELLIS-3D) benchmark datasets using both hand-annotated ground truth labels as well as labels predicted by the state-of-the-art deep learning models, showing improvement over recent comparable calibration approaches.