# Rethinking the Multi-view Stereo from the Perspective of Rendering-based Augmentation

arXiv: https://arxiv.org/abs/2303.06418 · HTML: https://mlsanity.com/paper/2303.06418/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 11 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Multi-Scale Geometric Consistency Guided Multi-View Stereo](https://mlsanity.com/paper/1904.08103/) (2019)
- [Planar Prior Assisted PatchMatch Multi-View Stereo](https://mlsanity.com/paper/1912.11744/) (2019)
- [MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth](https://mlsanity.com/paper/2208.02541/) (2022)
- [RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering](https://mlsanity.com/paper/2203.03949/) (2022)
- [nLMVS-Net: Deep Non-Lambertian Multi-View Stereo](https://mlsanity.com/paper/2207.11876/) (2022)

## Abstract (arXiv)

GigaMVS presents several challenges to existing Multi-View Stereo (MVS) algorithms for its large scale, complex occlusions, and gigapixel images. To address these problems, we first apply one of the state-of-the-art learning-based MVS methods, --MVSFormer, to overcome intractable scenarios such as textureless and reflections regions suffered by traditional PatchMatch methods, but it fails in a few large scenes' reconstructions. Moreover, traditional PatchMatch algorithms such as ACMMP, OpenMVS, and RealityCapture are leveraged to further improve the completeness in large scenes. Furthermore, to unify both advantages of deep learning methods and the traditional PatchMatch, we propose to render depth and color images to further fine-tune the MVSFormer model. Notably, we find that the MVS method could produce much better predictions through rendered images due to the coincident illumination, which we believe is significant for the MVS community. Thus, MVSFormer is capable of generalizing to large-scale scenes and complementarily solves the textureless reconstruction problem. Finally, we have assembled all point clouds mentioned above textitexcept ones from RealityCapture and ranked Top-1 on the competitive GigaReconstruction.