# NeuralFusion: Online Depth Fusion in Latent Space

arXiv: https://arxiv.org/abs/2011.14791 · HTML: https://mlsanity.com/paper/2011.14791/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 30 of 46)

- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [Point-SLAM: Dense Neural Point Cloud-based SLAM](https://mlsanity.com/paper/2304.04278/) (2023)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting](https://mlsanity.com/paper/2312.10070/) (2023)
- [Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussians](https://mlsanity.com/paper/2405.16544/) (2024)
- [Loopy-SLAM: Dense Neural SLAM with Loop Closures](https://mlsanity.com/paper/2402.09944/) (2024)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [GlORIE-SLAM: Globally Optimized RGB-only Implicit Encoding Point Cloud SLAM](https://mlsanity.com/paper/2403.19549/) (2024)
- [Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations](https://mlsanity.com/paper/2108.05851/) (2021)
- [BNV-Fusion: Dense 3D Reconstruction using Bi-level Neural Volume Fusion](https://mlsanity.com/paper/2204.01139/) (2022)
- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction](https://mlsanity.com/paper/2308.08741/) (2023)
- [DNS SLAM: Dense Neural Semantic-Informed SLAM](https://mlsanity.com/paper/2312.00204/) (2023)
- [UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM](https://mlsanity.com/paper/2306.11048/) (2023)
- [NIS-SLAM: Neural Implicit Semantic RGB-D SLAM for 3D Consistent Scene Understanding](https://mlsanity.com/paper/2407.20853/) (2024)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [Probabilistic Volumetric Fusion for Dense Monocular SLAM](https://mlsanity.com/paper/2210.01276/) (2022)
- [CVRecon: Rethinking 3D Geometric Feature Learning For Neural Reconstruction](https://mlsanity.com/paper/2304.14633/) (2023)
- ["Where am I?" Scene Retrieval with Language](https://mlsanity.com/paper/2404.14565/) (2024)
- [NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping](https://mlsanity.com/paper/2110.09415/) (2021)
- [Multi-sensor large-scale dataset for multi-view 3D reconstruction](https://mlsanity.com/paper/2203.06111/) (2022)
- [Gradient-SDF: A Semi-Implicit Surface Representation for 3D Reconstruction](https://mlsanity.com/paper/2111.13652/) (2021)
- [Temporally Consistent Online Depth Estimation Using Point-Based Fusion](https://mlsanity.com/paper/2304.07435/) (2023)
- [AGS-Mesh: Adaptive Gaussian Splatting and Meshing with Geometric Priors for Indoor Room Reconstruction Using Smartphones](https://mlsanity.com/paper/2411.19271/) (2024)
- [HRBF-Fusion: Accurate 3D reconstruction from RGB-D data using on-the-fly implicits](https://mlsanity.com/paper/2202.01829/) (2022)
- [Learning Online Multi-Sensor Depth Fusion](https://mlsanity.com/paper/2204.03353/) (2022)
- [LiveNVS: Neural View Synthesis on Live RGB-D Streams](https://mlsanity.com/paper/2311.16668/) (2023)
- [ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction](https://mlsanity.com/paper/2311.18068/) (2023)

## Abstract (arXiv)

We present a novel online depth map fusion approach that learns depth map aggregation in a latent feature space. While previous fusion methods use an explicit scene representation like signed distance functions (SDFs), we propose a learned feature representation for the fusion. The key idea is a separation between the scene representation used for the fusion and the output scene representation, via an additional translator network. Our neural network architecture consists of two main parts: a depth and feature fusion sub-network, which is followed by a translator sub-network to produce the final surface representation (e.g. TSDF) for visualization or other tasks. Our approach is an online process, handles high noise levels, and is particularly able to deal with gross outliers common for photometric stereo-based depth maps. Experiments on real and synthetic data demonstrate improved results compared to the state of the art, especially in challenging scenarios with large amounts of noise and outliers.