# PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility

arXiv: https://arxiv.org/abs/2108.08943 · HTML: https://mlsanity.com/paper/2108.08943/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Group-wise Correlation Stereo Network](https://mlsanity.com/paper/1903.04025/) (2019)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis](https://mlsanity.com/paper/1708.01749/) (2017)
- [DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch](https://mlsanity.com/paper/1909.05845/) (2019)
- [Multi-Scale Geometric Consistency Guided Multi-View Stereo](https://mlsanity.com/paper/1904.08103/) (2019)
- [Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement](https://mlsanity.com/paper/2003.13017/) (2020)
- [Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume](https://mlsanity.com/paper/1912.11746/) (2019)
- [Planar Prior Assisted PatchMatch Multi-View Stereo](https://mlsanity.com/paper/1912.11744/) (2019)
- [Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras](https://mlsanity.com/paper/1708.07878/) (2017)
- [PVSNet: Pixelwise Visibility-Aware Multi-View Stereo Network](https://mlsanity.com/paper/2007.07714/) (2020)
- [TAPA-MVS: Textureless-Aware PAtchMatch Multi-View Stereo](https://mlsanity.com/paper/1903.10929/) (2019)
- [DeepC-MVS: Deep Confidence Prediction for Multi-View Stereo Reconstruction](https://mlsanity.com/paper/1912.00439/) (2019)
- [BP-MVSNet: Belief-Propagation-Layers for Multi-View-Stereo](https://mlsanity.com/paper/2010.12436/) (2020)
- [Geometry-Informed Material Recognition](https://mlsanity.com/paper/1607.05338/) (2016)

## Built on by (most cited first, 17 of 17)

- [IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo](https://mlsanity.com/paper/2112.05126/) (2021)
- [MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth](https://mlsanity.com/paper/2208.02541/) (2022)
- [MVSTER: Epipolar Transformer for Efficient Multi-View Stereo](https://mlsanity.com/paper/2204.07346/) (2022)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization](https://mlsanity.com/paper/2401.06385/) (2024)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [Non-parametric Depth Distribution Modelling based Depth Inference for Multi-view Stereo](https://mlsanity.com/paper/2205.03783/) (2022)
- [CostFormer:Cost Transformer for Cost Aggregation in Multi-view Stereo](https://mlsanity.com/paper/2305.10320/) (2023)
- [TSAR-MVS: Textureless-aware Segmentation and Correlative Refinement Guided Multi-View Stereo](https://mlsanity.com/paper/2308.09990/) (2023)
- [Hierarchical Prior Mining for Non-local Multi-View Stereo](https://mlsanity.com/paper/2303.09758/) (2023)
- [RobustMVS: Single Domain Generalized Deep Multi-view Stereo](https://mlsanity.com/paper/2405.09131/) (2024)
- [SceneDiff: A Benchmark and Method for Multiview Object Change Detection](https://mlsanity.com/paper/2512.16908/) (2025)
- [Polarimetric PatchMatch Multi-View Stereo](https://mlsanity.com/paper/2311.07600/) (2023)
- [MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance](https://mlsanity.com/paper/2404.08252/) (2024)
- [Deep PatchMatch MVS with Learned Patch Coplanarity, Geometric Consistency and Adaptive Pixel Sampling](https://mlsanity.com/paper/2210.07582/) (2022)
- [Select-and-Combine (SAC): A Novel Multi-Stereo Depth Fusion Algorithm for Point Cloud Generation via Efficient Local Markov Netlets](https://mlsanity.com/paper/2308.12138/) (2023)
- [Learning Deformable Hypothesis Sampling for Accurate PatchMatch Multi-View Stereo](https://mlsanity.com/paper/2312.15970/) (2023)

## Abstract (arXiv)

Recent learning-based multi-view stereo (MVS) methods show excellent performance with dense cameras and small depth ranges. However, non-learning based approaches still outperform for scenes with large depth ranges and sparser wide-baseline views, in part due to their PatchMatch optimization over pixelwise estimates of depth, normals, and visibility. In this paper, we propose an end-to-end trainable PatchMatch-based MVS approach that combines advantages of trainable costs and regularizations with pixelwise estimates. To overcome the challenge of the non-differentiable PatchMatch optimization that involves iterative sampling and hard decisions, we use reinforcement learning to minimize expected photometric cost and maximize likelihood of ground truth depth and normals. We incorporate normal estimation by using dilated patch kernels, and propose a recurrent cost regularization that applies beyond frontal plane-sweep algorithms to our pixelwise depth/normal estimates. We evaluate our method on widely used MVS benchmarks, ETH3D and Tanks and Temples (TnT), and compare to other state of the art learning based MVS models. On ETH3D, our method outperforms other recent learning-based approaches and performs comparably on advanced TnT.