# NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo

arXiv: https://arxiv.org/abs/2109.01129 · HTML: https://mlsanity.com/paper/2109.01129/

## Key facts

- Cited by 192 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 65 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 30 of 44)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)

## Built on by (most cited first, 30 of 192)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization](https://mlsanity.com/paper/2303.07418/) (2023)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://mlsanity.com/paper/2211.11738/) (2022)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025)
- [Instant Volumetric Head Avatars](https://mlsanity.com/paper/2211.12499/) (2022)
- [NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)](https://mlsanity.com/paper/2210.00379/) (2022)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Efficient Neural Radiance Fields for Interactive Free-viewpoint Video](https://mlsanity.com/paper/2112.01517/) (2021)
- [DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing](https://mlsanity.com/paper/2403.17822/) (2024)
- [SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction](https://mlsanity.com/paper/2311.12754/) (2023)
- [NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors](https://mlsanity.com/paper/2206.13597/) (2022)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)
- [NeRFReN: Neural Radiance Fields with Reflections](https://mlsanity.com/paper/2111.15234/) (2021)
- [S-NeRF: Neural Radiance Fields for Street Views](https://mlsanity.com/paper/2303.00749/) (2023)
- [NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2203.11283/) (2022)
- [NeRF-SR: High-Quality Neural Radiance Fields using Supersampling](https://mlsanity.com/paper/2112.01759/) (2021)
- [Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields](https://mlsanity.com/paper/2309.03185/) (2023)
- [FlashGS: Efficient 3D Gaussian Splatting for Large-scale and High-resolution Rendering](https://mlsanity.com/paper/2408.07967/) (2024)
- [Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory](https://mlsanity.com/paper/2507.02863/) (2025)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [Conditional-Flow NeRF: Accurate 3D Modelling with Reliable Uncertainty Quantification](https://mlsanity.com/paper/2203.10192/) (2022)

## Abstract (arXiv)

In this work, we present a new multi-view depth estimation method that utilizes both conventional reconstruction and learning-based priors over the recently proposed neural radiance fields (NeRF). Unlike existing neural network based optimization method that relies on estimated correspondences, our method directly optimizes over implicit volumes, eliminating the challenging step of matching pixels in indoor scenes. The key to our approach is to utilize the learning-based priors to guide the optimization process of NeRF. Our system firstly adapts a monocular depth network over the target scene by finetuning on its sparse SfM+MVS reconstruction from COLMAP. Then, we show that the shape-radiance ambiguity of NeRF still exists in indoor environments and propose to address the issue by employing the adapted depth priors to monitor the sampling process of volume rendering. Finally, a per-pixel confidence map acquired by error computation on the rendered image can be used to further improve the depth quality. Experiments show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes, with surprising findings presented on the effectiveness of correspondence-based optimization and NeRF-based optimization over the adapted depth priors. In addition, we show that the guided optimization scheme does not sacrifice the original synthesis capability of neural radiance fields, improving the rendering quality on both seen and novel views. Code is available at https://github.com/weiyithu/NerfingMVS.