# Physics and semantic informed multi-sensor calibration via optimization theory and self-supervised learning

arXiv: https://arxiv.org/abs/2206.02856 · HTML: https://mlsanity.com/paper/2206.02856/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 80 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 30 of 34)

- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [PU-GAN: a Point Cloud Upsampling Adversarial Network](https://mlsanity.com/paper/1907.10844/) (2019)
- [High-Quality Self-Supervised Deep Image Denoising](https://mlsanity.com/paper/1901.10277/) (2019)
- [Self-supervised Augmentation Consistency for Adapting Semantic Segmentation](https://mlsanity.com/paper/2105.00097/) (2021)
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](https://mlsanity.com/paper/1910.12240/) (2019)
- [Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review](https://mlsanity.com/paper/2004.05224/) (2020)
- [PU-GCN: Point Cloud Upsampling using Graph Convolutional Networks](https://mlsanity.com/paper/1912.03264/) (2019)
- [You Only Learn One Representation: Unified Network for Multiple Tasks](https://mlsanity.com/paper/2105.04206/) (2021)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [PUGeo-Net: A Geometry-centric Network for 3D Point Cloud Upsampling](https://mlsanity.com/paper/2002.10277/) (2020)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [Selective Sensor Fusion for Neural Visual-Inertial Odometry](https://mlsanity.com/paper/1903.01534/) (2019)
- [Self-supervised Geometric Perception](https://mlsanity.com/paper/2103.03114/) (2021)
- [Non-Rigid Image Registration Using Self-Supervised Fully Convolutional Networks without Training Data](https://mlsanity.com/paper/1801.04012/) (2018)
- [Monocular Differentiable Rendering for Self-Supervised 3D Object Detection](https://mlsanity.com/paper/2009.14524/) (2020)
- [MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review](https://mlsanity.com/paper/2108.03004/) (2021)
- [Mix-and-Match Tuning for Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/1712.00661/) (2017)
- [Self-Supervised Camera Self-Calibration from Video](https://mlsanity.com/paper/2112.03325/) (2021)
- [DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow](https://mlsanity.com/paper/2203.09385/) (2022)
- [SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera](https://mlsanity.com/paper/2003.04260/) (2020)
- [CFNet: LiDAR-Camera Registration Using Calibration Flow Network](https://mlsanity.com/paper/2104.11907/) (2021)
- [Improvements to Target-Based 3D LiDAR to Camera Calibration](https://mlsanity.com/paper/1910.03126/) (2019)
- [Online Extrinsic Calibration based on Per-Sensor Ego-Motion Using Dual Quaternions](https://mlsanity.com/paper/2101.11440/) (2021)
- [SemCal: Semantic LiDAR-Camera Calibration using Neural MutualInformation Estimator](https://mlsanity.com/paper/2109.10270/) (2021)

## Built on by (most cited first, 4 of 4)

- [Artistic Intelligence: A Diffusion-Based Framework for High-Fidelity Landscape Painting Synthesis](https://mlsanity.com/paper/2407.17229/) (2024)
- [RLCNet: An end-to-end deep learning framework for simultaneous online calibration of LiDAR, RADAR, and Camera](https://mlsanity.com/paper/2512.08262/) (2025)
- [Physics-Constrained Denoising Autoencoders for Data-Scarce Wildfire UAV Sensing](https://mlsanity.com/paper/2601.11794/) (2026)
- [CLRNet: Targetless Extrinsic Calibration for Camera, Lidar and 4D Radar Using Deep Learning](https://mlsanity.com/paper/2603.15767/) (2026)

## Abstract (arXiv)

Achieving safe and reliable autonomous driving relies greatly on the ability to achieve an accurate and robust perception system; however, this cannot be fully realized without precisely calibrated sensors. Environmental and operational conditions as well as improper maintenance can produce calibration errors inhibiting sensor fusion and, consequently, degrading the perception performance. Traditionally, sensor calibration is performed in a controlled environment with one or more known targets. Such a procedure can only be carried out in between drives and requires manual operation; a tedious task if needed to be conducted on a regular basis. This sparked a recent interest in online targetless methods, capable of yielding a set of geometric transformations based on perceived environmental features, however, the required redundancy in sensing modalities makes this task even more challenging, as the features captured by each modality and their distinctiveness may vary. We present a holistic approach to performing joint calibration of a camera-lidar-radar trio. Leveraging prior knowledge and physical properties of these sensing modalities together with semantic information, we propose two targetless calibration methods within a cost minimization framework once via direct online optimization, and second via self-supervised learning (SSL).