# Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation

arXiv: https://arxiv.org/abs/2104.05374 · HTML: https://mlsanity.com/paper/2104.05374/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis](https://mlsanity.com/paper/1708.01749/) (2017)
- [Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement](https://mlsanity.com/paper/2003.13017/) (2020)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [SegStereo: Exploiting Semantic Information for Disparity Estimation](https://mlsanity.com/paper/1807.11699/) (2018)
- [Deep Feature Factorization For Concept Discovery](https://mlsanity.com/paper/1806.10206/) (2018)
- [Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume](https://mlsanity.com/paper/1912.11746/) (2019)
- [Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics](https://mlsanity.com/paper/1906.05717/) (2019)
- [Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency](https://mlsanity.com/paper/1905.02706/) (2019)
- [MVS^2: Deep Unsupervised Multi-view Stereo with Multi-View Symmetry](https://mlsanity.com/paper/1908.11526/) (2019)
- [M^3VSNet: Unsupervised Multi-metric Multi-view Stereo Network](https://mlsanity.com/paper/2005.00363/) (2020)
- [Real-Time Semantic Stereo Matching](https://mlsanity.com/paper/1910.00541/) (2019)
- [Open-World Stereo Video Matching with Deep RNN](https://mlsanity.com/paper/1808.03959/) (2018)

## Built on by (most cited first, 30 of 31)

- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [Improving neural implicit surfaces geometry with patch warping](https://mlsanity.com/paper/2112.09648/) (2021)
- [TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo](https://mlsanity.com/paper/2111.07418/) (2021)
- [Digging into Uncertainty in Self-supervised Multi-view Stereo](https://mlsanity.com/paper/2108.12966/) (2021)
- [RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering](https://mlsanity.com/paper/2203.03949/) (2022)
- [LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives](https://mlsanity.com/paper/2404.09748/) (2024)
- [KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo](https://mlsanity.com/paper/2207.10425/) (2022)
- [C2F2NeUS: Cascade Cost Frustum Fusion for High Fidelity and Generalizable Neural Surface Reconstruction](https://mlsanity.com/paper/2306.10003/) (2023)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [DebSDF: Delving into the Details and Bias of Neural Indoor Scene Reconstruction](https://mlsanity.com/paper/2308.15536/) (2023)
- [GenS: Generalizable Neural Surface Reconstruction from Multi-View Images](https://mlsanity.com/paper/2406.02495/) (2024)
- [Prior-Guided Multi-View 3D Head Reconstruction](https://mlsanity.com/paper/2107.04277/) (2021)
- [Deep Multi-View Stereo gone wild](https://mlsanity.com/paper/2104.15119/) (2021)
- [DS-MVSNet: Unsupervised Multi-view Stereo via Depth Synthesis](https://mlsanity.com/paper/2208.06674/) (2022)
- [Joint Depth Prediction and Semantic Segmentation with Multi-View SAM](https://mlsanity.com/paper/2311.00134/) (2023)
- [Semi-supervised Deep Multi-view Stereo](https://mlsanity.com/paper/2207.11699/) (2022)
- [NeuralRoom: Geometry-Constrained Neural Implicit Surfaces for Indoor Scene Reconstruction](https://mlsanity.com/paper/2210.06853/) (2022)
- [RobustMVS: Single Domain Generalized Deep Multi-view Stereo](https://mlsanity.com/paper/2405.09131/) (2024)
- [CL-MVSNet: Unsupervised Multi-view Stereo with Dual-level Contrastive Learning](https://mlsanity.com/paper/2503.08219/) (2025)
- [PatchMVSNet: Patch-wise Unsupervised Multi-View Stereo for Weakly-Textured Surface Reconstruction](https://mlsanity.com/paper/2203.02156/) (2022)
- [Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity](https://mlsanity.com/paper/2211.16905/) (2022)
- [ControLRM: Fast and Controllable 3D Generation via Large Reconstruction Model](https://mlsanity.com/paper/2410.09592/) (2024)
- [HR-NeuS: Recovering High-Frequency Surface Geometry via Neural Implicit Surfaces](https://mlsanity.com/paper/2302.06793/) (2023)
- [Improved Neural Radiance Fields Using Pseudo-depth and Fusion](https://mlsanity.com/paper/2308.03772/) (2023)
- [SMFormer: Empowering Self-supervised Stereo Matching via Foundation Models and Data Augmentation](https://mlsanity.com/paper/2604.10218/) (2026)
- [ES-MVSNet: Efficient Framework for End-to-end Self-supervised Multi-View Stereo](https://mlsanity.com/paper/2308.02191/) (2023)
- [Geometric Constraints in Deep Learning Frameworks: A Survey](https://mlsanity.com/paper/2403.12431/) (2024)
- [Improving 3D Finger Traits Recognition via Generalizable Neural Rendering](https://mlsanity.com/paper/2410.09582/) (2024)
- [Prism: Semi-Supervised Multi-View Stereo with Monocular Structure Priors](https://mlsanity.com/paper/2412.05771/) (2024)
- [Boosting Multi-View Stereo with Depth Foundation Model in the Absence of Real-World Labels](https://mlsanity.com/paper/2504.11845/) (2025)

## Abstract (arXiv)

Recent studies have witnessed that self-supervised methods based on view synthesis obtain clear progress on multi-view stereo (MVS). However, existing methods rely on the assumption that the corresponding points among different views share the same color, which may not always be true in practice. This may lead to unreliable self-supervised signal and harm the final reconstruction performance. To address the issue, we propose a framework integrated with more reliable supervision guided by semantic co-segmentation and data-augmentation. Specially, we excavate mutual semantic from multi-view images to guide the semantic consistency. And we devise effective data-augmentation mechanism which ensures the transformation robustness by treating the prediction of regular samples as pseudo ground truth to regularize the prediction of augmented samples. Experimental results on DTU dataset show that our proposed methods achieve the state-of-the-art performance among unsupervised methods, and even compete on par with supervised methods. Furthermore, extensive experiments on Tanks&Temples dataset demonstrate the effective generalization ability of the proposed method.