# CT-MVSNet: Efficient Multi-View Stereo with Cross-scale Transformer

arXiv: https://arxiv.org/abs/2312.08594 · HTML: https://mlsanity.com/paper/2312.08594/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [PatchmatchNet: Learned Multi-View Patchmatch Stereo](https://mlsanity.com/paper/2012.01411/) (2020)
- [Cost Volume Pyramid Based Depth Inference for Multi-View Stereo](https://mlsanity.com/paper/1912.08329/) (2019)
- [Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness](https://mlsanity.com/paper/1911.12012/) (2019)
- [MatchFormer: Interleaving Attention in Transformers for Feature Matching](https://mlsanity.com/paper/2203.09645/) (2022)
- [TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers](https://mlsanity.com/paper/2111.14600/) (2021)
- [Rethinking Depth Estimation for Multi-View Stereo: A Unified Representation](https://mlsanity.com/paper/2201.01501/) (2022)
- [AA-RMVSNet: Adaptive Aggregation Recurrent Multi-view Stereo Network](https://mlsanity.com/paper/2108.03824/) (2021)
- [MVSTER: Epipolar Transformer for Efficient Multi-View Stereo](https://mlsanity.com/paper/2204.07346/) (2022)
- [MVS2D: Efficient Multi-view Stereo via Attention-Driven 2D Convolutions](https://mlsanity.com/paper/2104.13325/) (2021)
- [Generalized Binary Search Network for Highly-Efficient Multi-View Stereo](https://mlsanity.com/paper/2112.02338/) (2021)
- [WT-MVSNet: Window-based Transformers for Multi-view Stereo](https://mlsanity.com/paper/2205.14319/) (2022)
- [RayMVSNet++: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo](https://mlsanity.com/paper/2307.10233/) (2023)

## Built on by (most cited first, 2 of 2)

- [ThermalNeRF: Thermal Radiance Fields](https://mlsanity.com/paper/2407.15337/) (2024)
- [3DTV: A Feedforward Interpolation Network for Real-Time View Synthesis](https://mlsanity.com/paper/2604.11211/) (2026)

## Abstract (arXiv)

Recent deep multi-view stereo (MVS) methods have widely incorporated transformers into cascade network for high-resolution depth estimation, achieving impressive results. However, existing transformer-based methods are constrained by their computational costs, preventing their extension to finer stages. In this paper, we propose a novel cross-scale transformer (CT) that processes feature representations at different stages without additional computation. Specifically, we introduce an adaptive matching-aware transformer (AMT) that employs different interactive attention combinations at multiple scales. This combined strategy enables our network to capture intra-image context information and enhance inter-image feature relationships. Besides, we present a dual-feature guided aggregation (DFGA) that embeds the coarse global semantic information into the finer cost volume construction to further strengthen global and local feature awareness. Meanwhile, we design a feature metric loss (FM Loss) that evaluates the feature bias before and after transformation to reduce the impact of feature mismatch on depth estimation. Extensive experiments on DTU dataset and Tanks and Temples (T&T) benchmark demonstrate that our method achieves state-of-the-art results. Code is available at https://github.com/wscstrive/CT-MVSNet.