# Self-supervised Learning of Depth Inference for Multi-view Stereo

arXiv: https://arxiv.org/abs/2104.02972 · HTML: https://mlsanity.com/paper/2104.02972/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 7 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [Visibility-aware Multi-view Stereo Network](https://mlsanity.com/paper/2008.07928/) (2020)
- [Pyramid Multi-view Stereo Net with Self-adaptive View Aggregation](https://mlsanity.com/paper/1912.03001/) (2019)
- [Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency](https://mlsanity.com/paper/1905.02706/) (2019)
- [PVSNet: Pixelwise Visibility-Aware Multi-View Stereo Network](https://mlsanity.com/paper/2007.07714/) (2020)
- [MVS^2: Deep Unsupervised Multi-view Stereo with Multi-View Symmetry](https://mlsanity.com/paper/1908.11526/) (2019)
- [M^3VSNet: Unsupervised Multi-metric Multi-view Stereo Network](https://mlsanity.com/paper/2005.00363/) (2020)
- [Learning to Adapt Multi-View Stereo by Self-Supervision](https://mlsanity.com/paper/2009.13278/) (2020)

## Built on by (most cited first, 28 of 28)

- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images](https://mlsanity.com/paper/2302.04233/) (2023)
- [RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering](https://mlsanity.com/paper/2203.03949/) (2022)
- [LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives](https://mlsanity.com/paper/2404.09748/) (2024)
- [MVSAnywhere: Zero-Shot Multi-View Stereo](https://mlsanity.com/paper/2503.22430/) (2025)
- [KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo](https://mlsanity.com/paper/2207.10425/) (2022)
- [C2F2NeUS: Cascade Cost Frustum Fusion for High Fidelity and Generalizable Neural Surface Reconstruction](https://mlsanity.com/paper/2306.10003/) (2023)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [PatchRefiner: Leveraging Synthetic Data for Real-Domain High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2406.06679/) (2024)
- [Non-parametric Depth Distribution Modelling based Depth Inference for Multi-view Stereo](https://mlsanity.com/paper/2205.03783/) (2022)
- [GenS: Generalizable Neural Surface Reconstruction from Multi-View Images](https://mlsanity.com/paper/2406.02495/) (2024)
- [DS-MVSNet: Unsupervised Multi-view Stereo via Depth Synthesis](https://mlsanity.com/paper/2208.06674/) (2022)
- [SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery](https://mlsanity.com/paper/2406.18151/) (2024)
- [3D-PL: Domain Adaptive Depth Estimation with 3D-aware Pseudo-Labeling](https://mlsanity.com/paper/2209.09231/) (2022)
- [CostFilter-AD: Enhancing Anomaly Detection through Matching Cost Filtering](https://mlsanity.com/paper/2505.01476/) (2025)
- [CL-MVSNet: Unsupervised Multi-view Stereo with Dual-level Contrastive Learning](https://mlsanity.com/paper/2503.08219/) (2025)
- [PatchMVSNet: Patch-wise Unsupervised Multi-View Stereo for Weakly-Textured Surface Reconstruction](https://mlsanity.com/paper/2203.02156/) (2022)
- [ES-MVSNet: Efficient Framework for End-to-end Self-supervised Multi-View Stereo](https://mlsanity.com/paper/2308.02191/) (2023)
- [Geometric Constraints in Deep Learning Frameworks: A Survey](https://mlsanity.com/paper/2403.12431/) (2024)
- [Fine-Grained Urban Flow Inference with Multi-scale Representation Learning](https://mlsanity.com/paper/2406.09710/) (2024)
- [PFDepth: Heterogeneous Pinhole-Fisheye Joint Depth Estimation via Distortion-aware Gaussian-Splatted Volumetric Fusion](https://mlsanity.com/paper/2509.26008/) (2025)
- [Exploring the Mutual Influence between Self-Supervised Single-Frame and Multi-Frame Depth Estimation](https://mlsanity.com/paper/2304.12685/) (2023)
- [Self-Supervised Learning for Point Clouds Data: A Survey](https://mlsanity.com/paper/2305.11881/) (2023)
- [Prism: Semi-Supervised Multi-View Stereo with Monocular Structure Priors](https://mlsanity.com/paper/2412.05771/) (2024)
- [Boosting Multi-View Stereo with Depth Foundation Model in the Absence of Real-World Labels](https://mlsanity.com/paper/2504.11845/) (2025)
- [MVS-TTA: Test-Time Adaptation for Multi-View Stereo via Meta-Auxiliary Learning](https://mlsanity.com/paper/2511.18120/) (2025)
- [GeoSurDepth: Harnessing Foundation Model for Spatial Geometry Consistency-Oriented Self-Supervised Surround-View Depth Estimation](https://mlsanity.com/paper/2601.05839/) (2026)
- [Cross-Vehicle 3D Geometric Consistency for Self-Supervised Surround Depth Estimation on Articulated Vehicles](https://mlsanity.com/paper/2604.02639/) (2026)

## Abstract (arXiv)

Recent supervised multi-view depth estimation networks have achieved promising results. Similar to all supervised approaches, these networks require ground-truth data during training. However, collecting a large amount of multi-view depth data is very challenging. Here, we propose a self-supervised learning framework for multi-view stereo that exploit pseudo labels from the input data. We start by learning to estimate depth maps as initial pseudo labels under an unsupervised learning framework relying on image reconstruction loss as supervision. We then refine the initial pseudo labels using a carefully designed pipeline leveraging depth information inferred from higher resolution images and neighboring views. We use these high-quality pseudo labels as the supervision signal to train the network and improve, iteratively, its performance by self-training. Extensive experiments on the DTU dataset show that our proposed self-supervised learning framework outperforms existing unsupervised multi-view stereo networks by a large margin and performs on par compared to the supervised counterpart. Code is available at https://github.com/JiayuYANG/Self-supervised-CVP-MVSNet.