# Learning Online Multi-Sensor Depth Fusion

arXiv: https://arxiv.org/abs/2204.03353 · HTML: https://mlsanity.com/paper/2204.03353/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 68 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning](https://mlsanity.com/paper/1611.03631/) (2016)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [Don't Forget The Past: Recurrent Depth Estimation from Monocular Video](https://mlsanity.com/paper/2001.02613/) (2020)
- [DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors](https://mlsanity.com/paper/2012.05551/) (2020)
- [RoutedFusion: Learning Real-time Depth Map Fusion](https://mlsanity.com/paper/2001.04388/) (2020)
- [NeuralFusion: Online Depth Fusion in Latent Space](https://mlsanity.com/paper/2011.14791/) (2020)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations](https://mlsanity.com/paper/2108.05851/) (2021)
- [PSDF Fusion: Probabilistic Signed Distance Function for On-the-fly 3D Data Fusion and Scene Reconstruction](https://mlsanity.com/paper/1807.11034/) (2018)
- [Fusion of stereo and still monocular depth estimates in a self-supervised learning context](https://mlsanity.com/paper/1803.07512/) (2018)
- [Collaborative Large-Scale Dense 3D Reconstruction with Online Inter-Agent Pose Optimisation](https://mlsanity.com/paper/1801.08361/) (2018)
- [Sdf-GAN: Semi-supervised Depth Fusion with Multi-scale Adversarial Networks](https://mlsanity.com/paper/1803.06657/) (2018)
- [Fusion of Range and Stereo Data for High-Resolution Scene-Modeling](https://mlsanity.com/paper/2012.06769/) (2020)
- [Learned Semantic Multi-Sensor Depth Map Fusion](https://mlsanity.com/paper/1909.00703/) (2019)

## Built on by (most cited first, 6 of 6)

- [Point-SLAM: Dense Neural Point Cloud-based SLAM](https://mlsanity.com/paper/2304.04278/) (2023)
- [Loopy-SLAM: Dense Neural SLAM with Loop Closures](https://mlsanity.com/paper/2402.09944/) (2024)
- [UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM](https://mlsanity.com/paper/2306.11048/) (2023)
- [Multi-View Photometric Stereo Revisited](https://mlsanity.com/paper/2210.07670/) (2022)
- [Mobile Robotic Multi-View Photometric Stereo](https://mlsanity.com/paper/2502.10842/) (2025)
- [A Multi-modal Garden Dataset and Hybrid 3D Dense Reconstruction Framework Based on Panoramic Stereo Images for a Trimming Robot](https://mlsanity.com/paper/2305.06278/) (2023)

## Abstract (arXiv)

Many hand-held or mixed reality devices are used with a single sensor for 3D reconstruction, although they often comprise multiple sensors. Multi-sensor depth fusion is able to substantially improve the robustness and accuracy of 3D reconstruction methods, but existing techniques are not robust enough to handle sensors which operate with diverse value ranges as well as noise and outlier statistics. To this end, we introduce SenFuNet, a depth fusion approach that learns sensor-specific noise and outlier statistics and combines the data streams of depth frames from different sensors in an online fashion. Our method fuses multi-sensor depth streams regardless of time synchronization and calibration and generalizes well with little training data. We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets, as well as the Replica dataset. Experiments demonstrate that our fusion strategy outperforms traditional and recent online depth fusion approaches. In addition, the combination of multiple sensors yields more robust outlier handling and more precise surface reconstruction than the use of a single sensor. The source code and data are available at https://github.com/tfy14esa/SenFuNet.