# OctNetFusion: Learning Depth Fusion from Data

arXiv: https://arxiv.org/abs/1704.01047 · HTML: https://mlsanity.com/paper/1704.01047/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2017.

## Built on by (most cited first, 30 of 45)

- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance](https://mlsanity.com/paper/2312.00846/) (2023)
- [OctFormer: Octree-based Transformers for 3D Point Clouds](https://mlsanity.com/paper/2305.03045/) (2023)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [POCO: Point Convolution for Surface Reconstruction](https://mlsanity.com/paper/2201.01831/) (2022)
- [3D Semantic Scene Completion: a Survey](https://mlsanity.com/paper/2103.07466/) (2021)
- [VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction](https://mlsanity.com/paper/2406.05774/) (2024)
- [Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling](https://mlsanity.com/paper/2112.04148/) (2021)
- [Multiresolution Deep Implicit Functions for 3D Shape Representation](https://mlsanity.com/paper/2109.05591/) (2021)
- [Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes](https://mlsanity.com/paper/2206.04942/) (2022)
- [GaussianRoom: Improving 3D Gaussian Splatting with SDF Guidance and Monocular Cues for Indoor Scene Reconstruction](https://mlsanity.com/paper/2405.19671/) (2024)
- [Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations](https://mlsanity.com/paper/2108.05851/) (2021)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [ALTO: Alternating Latent Topologies for Implicit 3D Reconstruction](https://mlsanity.com/paper/2212.04096/) (2022)
- [NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping](https://mlsanity.com/paper/2111.15207/) (2021)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [DeepDT: Learning Geometry From Delaunay Triangulation for Surface Reconstruction](https://mlsanity.com/paper/2101.10353/) (2021)
- [SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net](https://mlsanity.com/paper/2403.08885/) (2024)
- [Points2NeRF: Generating Neural Radiance Fields from 3D point cloud](https://mlsanity.com/paper/2206.01290/) (2022)
- [Pushing the Limits of 3D Shape Generation at Scale](https://mlsanity.com/paper/2306.11510/) (2023)
- [A Deep Signed Directional Distance Function for Object Shape Representation](https://mlsanity.com/paper/2107.11024/) (2021)
- [AGS-Mesh: Adaptive Gaussian Splatting and Meshing with Geometric Priors for Indoor Room Reconstruction Using Smartphones](https://mlsanity.com/paper/2411.19271/) (2024)
- [Multi-View Reconstruction using Signed Ray Distance Functions (SRDF)](https://mlsanity.com/paper/2209.00082/) (2022)
- [Unsupervised Shape Completion via Deep Prior in the Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2104.09023/) (2021)
- [PlaNeRF: SVD Unsupervised 3D Plane Regularization for NeRF Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2305.16914/) (2023)
- [Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability](https://mlsanity.com/paper/2402.12225/) (2024)
- [Point Cloud Upsampling and Normal Estimation using Deep Learning for Robust Surface Reconstruction](https://mlsanity.com/paper/2102.13391/) (2021)
- [LightSAL: Lightweight Sign Agnostic Learning for Implicit Surface Representation](https://mlsanity.com/paper/2103.14273/) (2021)

## Abstract (arXiv)

In this paper, we present a learning based approach to depth fusion, i.e., dense 3D reconstruction from multiple depth images. The most common approach to depth fusion is based on averaging truncated signed distance functions, which was originally proposed by Curless and Levoy in 1996. While this method is simple and provides great results, it is not able to reconstruct (partially) occluded surfaces and requires a large number frames to filter out sensor noise and outliers. Motivated by the availability of large 3D model repositories and recent advances in deep learning, we present a novel 3D CNN architecture that learns to predict an implicit surface representation from the input depth maps. Our learning based method significantly outperforms the traditional volumetric fusion approach in terms of noise reduction and outlier suppression. By learning the structure of real world 3D objects and scenes, our approach is further able to reconstruct occluded regions and to fill in gaps in the reconstruction. We demonstrate that our learning based approach outperforms both vanilla TSDF fusion as well as TV-L1 fusion on the task of volumetric fusion. Further, we demonstrate state-of-the-art 3D shape completion results.