# Calibrating LiDAR and Camera using Semantic Mutual information

arXiv: https://arxiv.org/abs/2104.12023 · HTML: https://mlsanity.com/paper/2104.12023/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 24 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [RELLIS-3D Dataset: Data, Benchmarks and Analysis](https://mlsanity.com/paper/2011.12954/) (2020)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer](https://mlsanity.com/paper/1511.03240/) (2015)
- [CCMI : Classifier based Conditional Mutual Information Estimation](https://mlsanity.com/paper/1906.01824/) (2019)
- [SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera](https://mlsanity.com/paper/2003.04260/) (2020)
- [CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks](https://mlsanity.com/paper/2103.14793/) (2021)
- [C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation](https://mlsanity.com/paper/2005.08226/) (2020)
- [DRMIME: Differentiable Mutual Information and Matrix Exponential for Multi-Resolution Image Registration](https://mlsanity.com/paper/2001.09865/) (2020)
- [Extrinsic Calibration of a 3D-LIDAR and a Camera](https://mlsanity.com/paper/2003.01213/) (2020)
- [LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network](https://mlsanity.com/paper/2012.13901/) (2020)
- [Experimental Evaluation of 3D-LIDAR Camera Extrinsic Calibration](https://mlsanity.com/paper/2007.01959/) (2020)

## Built on by (most cited first, 1 of 1)

- [INF: Implicit Neural Fusion for LiDAR and Camera](https://mlsanity.com/paper/2308.14414/) (2023)

## Abstract (arXiv)

We propose an algorithm for automatic, targetless, extrinsic calibration of a LiDAR and camera system using semantic information. We achieve this goal by maximizing mutual information (MI) of semantic information between sensors, leveraging a neural network to estimate semantic mutual information, and matrix exponential for calibration computation. Using kernel-based sampling to sample data from camera measurement based on LiDAR projected points, we formulate the problem as a novel differentiable objective function which supports the use of gradient-based optimization methods. We also introduce an initial calibration method using 2D MI-based image registration. Finally, we demonstrate the robustness of our method and quantitatively analyze the accuracy on a synthetic dataset and also evaluate our algorithm qualitatively on KITTI360 and RELLIS-3D benchmark datasets, showing improvement over recent comparable approaches.