# Neural 3D Scene Reconstruction from Multiple 2D Images without 3D Supervision

arXiv: https://arxiv.org/abs/2306.17643 · HTML: https://mlsanity.com/paper/2306.17643/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 48 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [SAL: Sign Agnostic Learning of Shapes from Raw Data](https://mlsanity.com/paper/1911.10414/) (2019)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [PatchmatchNet: Learned Multi-View Patchmatch Stereo](https://mlsanity.com/paper/2012.01411/) (2020)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo](https://mlsanity.com/paper/2109.01129/) (2021)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors](https://mlsanity.com/paper/2206.13597/) (2022)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)

## Built on by (most cited first, 2 of 2)

- [3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V](https://mlsanity.com/paper/2312.09738/) (2023)
- [FreBIS: Frequency-Based Stratification for Neural Implicit Surface Representations](https://mlsanity.com/paper/2504.20222/) (2025)

## Abstract (arXiv)

Neural 3D scene reconstruction methods have achieved impressive performance when reconstructing complex geometry and low-textured regions in indoor scenes. However, these methods heavily rely on 3D data which is costly and time-consuming to obtain in real world. In this paper, we propose a novel neural reconstruction method that reconstructs scenes using sparse depth under the plane constraints without 3D supervision. We introduce a signed distance function field, a color field, and a probability field to represent a scene. We optimize these fields to reconstruct the scene by using differentiable ray marching with accessible 2D images as supervision. We improve the reconstruction quality of complex geometry scene regions with sparse depth obtained by using the geometric constraints. The geometric constraints project 3D points on the surface to similar-looking regions with similar features in different 2D images. We impose the plane constraints to make large planes parallel or vertical to the indoor floor. Both two constraints help reconstruct accurate and smooth geometry structures of the scene. Without 3D supervision, our method achieves competitive performance compared with existing methods that use 3D supervision on the ScanNet dataset.