# AA-RMVSNet: Adaptive Aggregation Recurrent Multi-view Stereo Network

arXiv: https://arxiv.org/abs/2108.03824 · HTML: https://mlsanity.com/paper/2108.03824/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis](https://mlsanity.com/paper/1708.01749/) (2017)
- [Multi-Scale Geometric Consistency Guided Multi-View Stereo](https://mlsanity.com/paper/1904.08103/) (2019)
- [Visibility-aware Multi-view Stereo Network](https://mlsanity.com/paper/2008.07928/) (2020)
- [Dense Hybrid Recurrent Multi-view Stereo Net with Dynamic Consistency Checking](https://mlsanity.com/paper/2007.10872/) (2020)
- [Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume](https://mlsanity.com/paper/1912.11746/) (2019)
- [Pyramid Multi-view Stereo Net with Self-adaptive View Aggregation](https://mlsanity.com/paper/1912.03001/) (2019)
- [DeepC-MVS: Deep Confidence Prediction for Multi-View Stereo Reconstruction](https://mlsanity.com/paper/1912.00439/) (2019)

## Built on by (most cited first, 30 of 71)

- [Iterative Geometry Encoding Volume for Stereo Matching](https://mlsanity.com/paper/2303.06615/) (2023)
- [TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers](https://mlsanity.com/paper/2111.14600/) (2021)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereo](https://mlsanity.com/paper/2209.10248/) (2022)
- [Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation](https://mlsanity.com/paper/2201.01501/) (2022)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth](https://mlsanity.com/paper/2208.02541/) (2022)
- [STS: Surround-view Temporal Stereo for Multi-view 3D Detection](https://mlsanity.com/paper/2208.10145/) (2022)
- [Multiview Stereo with Cascaded Epipolar RAFT](https://mlsanity.com/paper/2205.04502/) (2022)
- [MVSTER: Epipolar Transformer for Efficient Multi-View Stereo](https://mlsanity.com/paper/2204.07346/) (2022)
- [MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo](https://mlsanity.com/paper/2401.11673/) (2024)
- [UniWorld: Autonomous Driving Pre-training via World Models](https://mlsanity.com/paper/2308.07234/) (2023)
- [Generalized Binary Search Network for Highly-Efficient Multi-View Stereo](https://mlsanity.com/paper/2112.02338/) (2021)
- [RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering](https://mlsanity.com/paper/2203.03949/) (2022)
- [S-VolSDF: Sparse Multi-View Stereo Regularization of Neural Implicit Surfaces](https://mlsanity.com/paper/2303.17712/) (2023)
- [WT-MVSNet: Window-based Transformers for Multi-view Stereo](https://mlsanity.com/paper/2205.14319/) (2022)
- [SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization](https://mlsanity.com/paper/2401.06385/) (2024)
- [KD-MVS: Knowledge Distillation Based Self-supervised Learning for Multi-view Stereo](https://mlsanity.com/paper/2207.10425/) (2022)
- [C2F2NeUS: Cascade Cost Frustum Fusion for High Fidelity and Generalizable Neural Surface Reconstruction](https://mlsanity.com/paper/2306.10003/) (2023)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [MSP-MVS: Multi-Granularity Segmentation Prior Guided Multi-View Stereo](https://mlsanity.com/paper/2407.19323/) (2024)
- [Non-parametric Depth Distribution Modelling based Depth Inference for Multi-view Stereo](https://mlsanity.com/paper/2205.03783/) (2022)
- [UniScene: Multi-Camera Unified Pre-training via 3D Scene Reconstruction for Autonomous Driving](https://mlsanity.com/paper/2305.18829/) (2023)
- [RayMVSNet++: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo](https://mlsanity.com/paper/2307.10233/) (2023)
- [Multi-View Stereo with Transformer](https://mlsanity.com/paper/2112.00336/) (2021)
- [Multi-sensor large-scale dataset for multi-view 3D reconstruction](https://mlsanity.com/paper/2203.06111/) (2022)
- [When Epipolar Constraint Meets Non-local Operators in Multi-View Stereo](https://mlsanity.com/paper/2309.17218/) (2023)
- [GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement](https://mlsanity.com/paper/2406.05649/) (2024)
- [Wheat3DGS: In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting](https://mlsanity.com/paper/2504.06978/) (2025)
- [SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint](https://mlsanity.com/paper/2503.13721/) (2025)

## Abstract (arXiv)

In this paper, we present a novel recurrent multi-view stereo network based on long short-term memory (LSTM) with adaptive aggregation, namely AA-RMVSNet. We firstly introduce an intra-view aggregation module to adaptively extract image features by using context-aware convolution and multi-scale aggregation, which efficiently improves the performance on challenging regions, such as thin objects and large low-textured surfaces. To overcome the difficulty of varying occlusion in complex scenes, we propose an inter-view cost volume aggregation module for adaptive pixel-wise view aggregation, which is able to preserve better-matched pairs among all views. The two proposed adaptive aggregation modules are lightweight, effective and complementary regarding improving the accuracy and completeness of 3D reconstruction. Instead of conventional 3D CNNs, we utilize a hybrid network with recurrent structure for cost volume regularization, which allows high-resolution reconstruction and finer hypothetical plane sweep. The proposed network is trained end-to-end and achieves excellent performance on various datasets. It ranks 1st among all submissions on Tanks and Temples benchmark and achieves competitive results on DTU dataset, which exhibits strong generalizability and robustness. Implementation of our method is available at https://github.com/QT-Zhu/AA-RMVSNet.