# MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth

arXiv: https://arxiv.org/abs/2208.02541 · HTML: https://mlsanity.com/paper/2208.02541/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 85 references that are in the map.
- First posted on arXiv 4 Aug 2022.

## Builds on (most cited first, 30 of 62)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases](https://mlsanity.com/paper/2103.10697/) (2021)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)

## Built on by (most cited first, 30 of 53)

- [HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction](https://mlsanity.com/paper/2410.06245/) (2024)
- [MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo](https://mlsanity.com/paper/2401.11673/) (2024)
- [MVSAnywhere: Zero-Shot Multi-View Stereo](https://mlsanity.com/paper/2503.22430/) (2025)
- [Multi-view Reconstruction via SfM-guided Monocular Depth Estimation](https://mlsanity.com/paper/2503.14483/) (2025)
- [MonSter++: Unified Stereo Matching, Multi-view Stereo, and Real-time Stereo with Monodepth Priors](https://mlsanity.com/paper/2501.08643/) (2025)
- [LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Merging](https://mlsanity.com/paper/2512.04939/) (2025)
- [GoMVS: Geometrically Consistent Cost Aggregation for Multi-View Stereo](https://mlsanity.com/paper/2404.07992/) (2024)
- [SparseRecon: Neural Implicit Surface Reconstruction from Sparse Views with Feature and Depth Consistencies](https://mlsanity.com/paper/2508.00366/) (2025)
- [Constraining Depth Map Geometry for Multi-View Stereo: A Dual-Depth Approach with Saddle-shaped Depth Cells](https://mlsanity.com/paper/2307.09160/) (2023)
- [FoVA-Depth: Field-of-View Agnostic Depth Estimation for Cross-Dataset Generalization](https://mlsanity.com/paper/2401.13786/) (2024)
- [LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion](https://mlsanity.com/paper/2507.05678/) (2025)
- [Multi-tiling Neural Radiance Field (NeRF) -- Geometric Assessment on Large-scale Aerial Datasets](https://mlsanity.com/paper/2310.00530/) (2023)
- [CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement](https://mlsanity.com/paper/2404.02225/) (2024)
- [Pixel-Aligned Multi-View Generation with Depth Guided Decoder](https://mlsanity.com/paper/2408.14016/) (2024)
- [SDL-MVS: View Space and Depth Deformable Learning Paradigm for Multi-View Stereo Reconstruction in Remote Sensing](https://mlsanity.com/paper/2405.17140/) (2024)
- [Improving Neural Surface Reconstruction with Feature Priors from Multi-View Image](https://mlsanity.com/paper/2408.02079/) (2024)
- [A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding](https://mlsanity.com/paper/2411.01893/) (2024)
- [RoMeO: Robust Metric Visual Odometry](https://mlsanity.com/paper/2412.11530/) (2024)
- [MonoMVSNet: Monocular Priors Guided Multi-View Stereo Network](https://mlsanity.com/paper/2507.11333/) (2025)
- [SRSplat: Feed-Forward Super-Resolution Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2511.12040/) (2025)
- [Geometry-Consistent 4D Gaussian Splatting for Sparse-Input Dynamic View Synthesis](https://mlsanity.com/paper/2511.23044/) (2025)
- [Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting](https://mlsanity.com/paper/2603.02893/) (2026)
- [ARAI-MVSNet: A multi-view stereo depth estimation network with adaptive depth range and depth interval](https://mlsanity.com/paper/2308.09022/) (2023)
- [Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs](https://mlsanity.com/paper/2403.04114/) (2024)
- [MVS-GS: High-Quality 3D Gaussian Splatting Mapping via Online Multi-View Stereo](https://mlsanity.com/paper/2412.19130/) (2024)
- [ICG-MVSNet: Learning Intra-view and Cross-view Relationships for Guidance in Multi-View Stereo](https://mlsanity.com/paper/2503.21525/) (2025)
- [Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimation](https://mlsanity.com/paper/2507.11540/) (2025)
- [MVSMamba: Multi-View Stereo with State Space Model](https://mlsanity.com/paper/2511.01315/) (2025)
- [LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving](https://mlsanity.com/paper/2603.03765/) (2026)
- [P2CADNet: An End-to-End Reconstruction Network for Parametric 3D CAD Model from Point Clouds](https://mlsanity.com/paper/2310.02638/) (2023)

## Abstract (arXiv)

Feature representation learning is the key recipe for learning-based Multi-View Stereo (MVS). As the common feature extractor of learning-based MVS, vanilla Feature Pyramid Networks (FPNs) suffer from discouraged feature representations for reflection and texture-less areas, which limits the generalization of MVS. Even FPNs worked with pre-trained Convolutional Neural Networks (CNNs) fail to tackle these issues. On the other hand, Vision Transformers (ViTs) have achieved prominent success in many 2D vision tasks. Thus we ask whether ViTs can facilitate feature learning in MVS? In this paper, we propose a pre-trained ViT enhanced MVS network called MVSFormer, which can learn more reliable feature representations benefited by informative priors from ViT. The finetuned MVSFormer with hierarchical ViTs of efficient attention mechanisms can achieve prominent improvement based on FPNs. Besides, the alternative MVSFormer with frozen ViT weights is further proposed. This largely alleviates the training cost with competitive performance strengthened by the attention map from the self-distillation pre-training. MVSFormer can be generalized to various input resolutions with efficient multi-scale training strengthened by gradient accumulation. Moreover, we discuss the merits and drawbacks of classification and regression-based MVS methods, and further propose to unify them with a temperature-based strategy. MVSFormer achieves state-of-the-art performance on the DTU dataset. Particularly, MVSFormer ranks as Top-1 on both intermediate and advanced sets of the highly competitive Tanks-and-Temples leaderboard.