# VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction

arXiv: https://arxiv.org/abs/2108.08623 · HTML: https://mlsanity.com/paper/2108.08623/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis](https://mlsanity.com/paper/1708.01749/) (2017)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion](https://mlsanity.com/paper/2012.02177/) (2020)
- [MVDepthNet: Real-time Multiview Depth Estimation Neural Network](https://mlsanity.com/paper/1807.08563/) (2018)
- [RoutedFusion: Learning Real-time Depth Map Fusion](https://mlsanity.com/paper/2001.04388/) (2020)
- [Normal Assisted Stereo Depth Estimation](https://mlsanity.com/paper/1911.10444/) (2019)
- [Multi-View Stereo by Temporal Nonparametric Fusion](https://mlsanity.com/paper/1904.06397/) (2019)
- [NeuralFusion: Online Depth Fusion in Latent Space](https://mlsanity.com/paper/2011.14791/) (2020)
- [RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials](https://mlsanity.com/paper/1901.01535/) (2019)
- [Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation](https://mlsanity.com/paper/2103.12964/) (2021)
- [Stereo Object Matching Network](https://mlsanity.com/paper/2103.12498/) (2021)

## Built on by (most cited first, 30 of 44)

- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Point-SLAM: Dense Neural Point Cloud-based SLAM](https://mlsanity.com/paper/2304.04278/) (2023)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting](https://mlsanity.com/paper/2312.10070/) (2023)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [PLGSLAM: Progressive Neural Scene Represenation with Local to Global Bundle Adjustment](https://mlsanity.com/paper/2312.09866/) (2023)
- [Loopy-SLAM: Dense Neural SLAM with Loop Closures](https://mlsanity.com/paper/2402.09944/) (2024)
- [NeuSurf: On-Surface Priors for Neural Surface Reconstruction from Sparse Input Views](https://mlsanity.com/paper/2312.13977/) (2023)
- [FatesGS: Fast and Accurate Sparse-View Surface Reconstruction using Gaussian Splatting with Depth-Feature Consistency](https://mlsanity.com/paper/2501.04628/) (2025)
- [NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising](https://mlsanity.com/paper/2403.20034/) (2024)
- [MonoNeuralFusion: Online Monocular Neural 3D Reconstruction with Geometric Priors](https://mlsanity.com/paper/2209.15153/) (2022)
- [FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction](https://mlsanity.com/paper/2304.01480/) (2023)
- [Deep Point Cloud Reconstruction](https://mlsanity.com/paper/2111.11704/) (2021)
- [Incremental Joint Learning of Depth, Pose and Implicit Scene Representation on Monocular Camera in Large-scale Scenes](https://mlsanity.com/paper/2404.06050/) (2024)
- [Multi-sensor large-scale dataset for multi-view 3D reconstruction](https://mlsanity.com/paper/2203.06111/) (2022)
- [Adaptive Surface Normal Constraint for Geometric Estimation from Monocular Images](https://mlsanity.com/paper/2402.05869/) (2024)
- [DORNet: A Degradation Oriented and Regularized Network for Blind Depth Super-Resolution](https://mlsanity.com/paper/2410.11666/) (2024)
- [Neural Scene Representation for Locomotion on Structured Terrain](https://mlsanity.com/paper/2206.08077/) (2022)
- [Temporally Consistent Online Depth Estimation Using Point-Based Fusion](https://mlsanity.com/paper/2304.07435/) (2023)
- [DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation](https://mlsanity.com/paper/2511.23127/) (2025)
- [VisFusion: Visibility-aware Online 3D Scene Reconstruction from Videos](https://mlsanity.com/paper/2304.10687/) (2023)
- [Learning Continuous Depth Representation via Geometric Spatial Aggregator](https://mlsanity.com/paper/2212.03499/) (2022)
- [CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoors Object Detection from Multi-view Images](https://mlsanity.com/paper/2403.04198/) (2024)
- [Learning Online Multi-Sensor Depth Fusion](https://mlsanity.com/paper/2204.03353/) (2022)
- [Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior](https://mlsanity.com/paper/2502.06338/) (2025)
- [Unified Scene Representation and Reconstruction for 3D Large Language Models](https://mlsanity.com/paper/2404.13044/) (2024)
- [PRISM-DP: Spatial Pose-based Observations for Diffusion-Policies via Segmentation, Mesh Generation, and Pose Tracking](https://mlsanity.com/paper/2504.20359/) (2025)
- [Range-Agnostic Multi-View Depth Estimation With Keyframe Selection](https://mlsanity.com/paper/2401.14401/) (2024)
- [Modern Augmented Reality: Applications, Trends, and Future Directions](https://mlsanity.com/paper/2202.09450/) (2022)

## Abstract (arXiv)

To reconstruct a 3D scene from a set of calibrated views, traditional multi-view stereo techniques rely on two distinct stages: local depth maps computation and global depth maps fusion. Recent studies concentrate on deep neural architectures for depth estimation by using conventional depth fusion method or direct 3D reconstruction network by regressing Truncated Signed Distance Function (TSDF). In this paper, we advocate that replicating the traditional two stages framework with deep neural networks improves both the interpretability and the accuracy of the results. As mentioned, our network operates in two steps: 1) the local computation of the local depth maps with a deep MVS technique, and, 2) the depth maps and images' features fusion to build a single TSDF volume. In order to improve the matching performance between images acquired from very different viewpoints (e.g., large-baseline and rotations), we introduce a rotation-invariant 3D convolution kernel called PosedConv. The effectiveness of the proposed architecture is underlined via a large series of experiments conducted on the ScanNet dataset where our approach compares favorably against both traditional and deep learning techniques.