# Digging into Uncertainty in Self-supervised Multi-view Stereo

arXiv: https://arxiv.org/abs/2108.12966 · HTML: https://mlsanity.com/paper/2108.12966/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 24 of 24)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [PatchmatchNet: Learned Multi-View Patchmatch Stereo](https://mlsanity.com/paper/2012.01411/) (2020)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis](https://mlsanity.com/paper/1708.01749/) (2017)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement](https://mlsanity.com/paper/2003.13017/) (2020)
- [Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume](https://mlsanity.com/paper/1912.11746/) (2019)
- [Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency](https://mlsanity.com/paper/1905.02706/) (2019)
- [Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation](https://mlsanity.com/paper/2104.05374/) (2021)
- [MVS^2: Deep Unsupervised Multi-view Stereo with Multi-View Symmetry](https://mlsanity.com/paper/1908.11526/) (2019)
- [M^3VSNet: Unsupervised Multi-metric Multi-view Stereo Network](https://mlsanity.com/paper/2005.00363/) (2020)
- [DeepC-MVS: Deep Confidence Prediction for Multi-View Stereo Reconstruction](https://mlsanity.com/paper/1912.00439/) (2019)
- [Learning to Adapt Multi-View Stereo by Self-Supervision](https://mlsanity.com/paper/2009.13278/) (2020)
- [Using Self-Contradiction to Learn Confidence Measures in Stereo Vision](https://mlsanity.com/paper/1604.05132/) (2016)

## Built on by (most cited first, 30 of 31)

- [Conditional-Flow NeRF: Accurate 3D Modelling with Reliable Uncertainty Quantification](https://mlsanity.com/paper/2203.10192/) (2022)
- [PlaneDepth: Self-supervised Depth Estimation via Orthogonal Planes](https://mlsanity.com/paper/2210.01612/) (2022)
- [RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering](https://mlsanity.com/paper/2203.03949/) (2022)
- [LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives](https://mlsanity.com/paper/2404.09748/) (2024)
- [Representation Uncertainty in Self-Supervised Learning as Variational Inference](https://mlsanity.com/paper/2203.11437/) (2022)
- [UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM](https://mlsanity.com/paper/2306.11048/) (2023)
- [KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo](https://mlsanity.com/paper/2207.10425/) (2022)
- [C2F2NeUS: Cascade Cost Frustum Fusion for High Fidelity and Generalizable Neural Surface Reconstruction](https://mlsanity.com/paper/2306.10003/) (2023)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [RayMVSNet++: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo](https://mlsanity.com/paper/2307.10233/) (2023)
- [GenS: Generalizable Neural Surface Reconstruction from Multi-View Images](https://mlsanity.com/paper/2406.02495/) (2024)
- [Robust RGB-D Fusion for Saliency Detection](https://mlsanity.com/paper/2208.01762/) (2022)
- [2D GANs Meet Unsupervised Single-view 3D Reconstruction](https://mlsanity.com/paper/2207.10183/) (2022)
- [DS-MVSNet: Unsupervised Multi-view Stereo via Depth Synthesis](https://mlsanity.com/paper/2208.06674/) (2022)
- [MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priors](https://mlsanity.com/paper/2303.03679/) (2023)
- [Semi-supervised Deep Multi-view Stereo](https://mlsanity.com/paper/2207.11699/) (2022)
- [RobustMVS: Single Domain Generalized Deep Multi-view Stereo](https://mlsanity.com/paper/2405.09131/) (2024)
- [CL-MVSNet: Unsupervised Multi-view Stereo with Dual-level Contrastive Learning](https://mlsanity.com/paper/2503.08219/) (2025)
- [PatchMVSNet: Patch-wise Unsupervised Multi-View Stereo for Weakly-Textured Surface Reconstruction](https://mlsanity.com/paper/2203.02156/) (2022)
- [ControLRM: Fast and Controllable 3D Generation via Large Reconstruction Model](https://mlsanity.com/paper/2410.09592/) (2024)
- [Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objects](https://mlsanity.com/paper/2212.14276/) (2022)
- [ViM-Disparity: Bridging the Gap of Speed, Accuracy and Memory for Disparity Map Generation](https://mlsanity.com/paper/2412.16745/) (2024)
- [RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo](https://mlsanity.com/paper/2204.01320/) (2022)
- [ES-MVSNet: Efficient Framework for End-to-end Self-supervised Multi-View Stereo](https://mlsanity.com/paper/2308.02191/) (2023)
- [Geometric Constraints in Deep Learning Frameworks: A Survey](https://mlsanity.com/paper/2403.12431/) (2024)
- [Improving 3D Finger Traits Recognition via Generalizable Neural Rendering](https://mlsanity.com/paper/2410.09582/) (2024)
- [DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality](https://mlsanity.com/paper/2511.12671/) (2025)
- [Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model](https://mlsanity.com/paper/2111.11055/) (2021)
- [Prism: Semi-Supervised Multi-View Stereo with Monocular Structure Priors](https://mlsanity.com/paper/2412.05771/) (2024)
- [Boosting Multi-View Stereo with Depth Foundation Model in the Absence of Real-World Labels](https://mlsanity.com/paper/2504.11845/) (2025)

## Abstract (arXiv)

Self-supervised Multi-view stereo (MVS) with a pretext task of image reconstruction has achieved significant progress recently. However, previous methods are built upon intuitions, lacking comprehensive explanations about the effectiveness of the pretext task in self-supervised MVS. To this end, we propose to estimate epistemic uncertainty in self-supervised MVS, accounting for what the model ignores. Specially, the limitations can be categorized into two types: ambiguious supervision in foreground and invalid supervision in background. To address these issues, we propose a novel Uncertainty reduction Multi-view Stereo (UMVS) framework for self-supervised learning. To alleviate ambiguous supervision in foreground, we involve extra correspondence prior with a flow-depth consistency loss. The dense 2D correspondence of optical flows is used to regularize the 3D stereo correspondence in MVS. To handle the invalid supervision in background, we use Monte-Carlo Dropout to acquire the uncertainty map and further filter the unreliable supervision signals on invalid regions. Extensive experiments on DTU and Tank&Temples benchmark show that our U-MVS framework achieves the best performance among unsupervised MVS methods, with competitive performance with its supervised opponents.