# INF: Implicit Neural Fusion for LiDAR and Camera

arXiv: https://arxiv.org/abs/2308.14414 · HTML: https://mlsanity.com/paper/2308.14414/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [General, Single-shot, Target-less, and Automatic LiDAR-Camera Extrinsic Calibration Toolbox](https://mlsanity.com/paper/2302.05094/) (2023)
- [OpenCalib: A Multi-sensor Calibration Toolbox for Autonomous Driving](https://mlsanity.com/paper/2205.14087/) (2022)
- [LiDAR and Camera Calibration using Motion Estimated by Sensor Fusion Odometry](https://mlsanity.com/paper/1804.05178/) (2018)
- [Automatic extrinsic calibration between a camera and a 3D Lidar using 3D point and plane correspondences](https://mlsanity.com/paper/1904.12433/) (2019)
- [Reflectance Intensity Assisted Automatic and Accurate Extrinsic Calibration of 3D LiDAR and Panoramic Camera Using a Printed Chessboard](https://mlsanity.com/paper/1708.05514/) (2017)
- [Experimental Evaluation of 3D-LIDAR Camera Extrinsic Calibration](https://mlsanity.com/paper/2007.01959/) (2020)
- [Calibrating LiDAR and Camera using Semantic Mutual information](https://mlsanity.com/paper/2104.12023/) (2021)

## Built on by (most cited first, 10 of 10)

- [SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields](https://mlsanity.com/paper/2311.15803/) (2023)
- [3DGS-Calib: 3D Gaussian Splatting for Multimodal SpatioTemporal Calibration](https://mlsanity.com/paper/2403.11577/) (2024)
- [UniCal: Unified Neural Sensor Calibration](https://mlsanity.com/paper/2409.18953/) (2024)
- [NeAS: 3D Reconstruction from X-ray Images using Neural Attenuation Surface](https://mlsanity.com/paper/2503.07491/) (2025)
- [REF2-NeRF: Reflection and Refraction aware Neural Radiance Field](https://mlsanity.com/paper/2311.17116/) (2023)
- [Robust LiDAR-Camera Calibration with 2D Gaussian Splatting](https://mlsanity.com/paper/2504.00525/) (2025)
- [Targetless LiDAR-Camera Calibration with Neural Gaussian Splatting](https://mlsanity.com/paper/2504.04597/) (2025)
- [NeRF-enabled Analysis-Through-Synthesis for ISAR Imaging of Small Everyday Objects with Sparse and Noisy UWB Radar Data](https://mlsanity.com/paper/2410.10085/) (2024)
- [PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views](https://mlsanity.com/paper/2606.27071/) (2026)
- [Geometry-Preserving in 3D Gaussian Splatting for LiDAR-Camera Extrinsic Calibration](https://mlsanity.com/paper/2606.20103/) (2026)

## Abstract (arXiv)

Sensor fusion has become a popular topic in robotics. However, conventional fusion methods encounter many difficulties, such as data representation differences, sensor variations, and extrinsic calibration. For example, the calibration methods used for LiDAR-camera fusion often require manual operation and auxiliary calibration targets. Implicit neural representations (INRs) have been developed for 3D scenes, and the volume density distribution involved in an INR unifies the scene information obtained by different types of sensors. Therefore, we propose implicit neural fusion (INF) for LiDAR and camera. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field. Along with the training process, INF both estimates LiDAR poses and optimizes extrinsic parameters. Our experiments demonstrate the high accuracy and stable performance of the proposed method.