# MatchFormer: Interleaving Attention in Transformers for Feature Matching

arXiv: https://arxiv.org/abs/2203.09645 · HTML: https://mlsanity.com/paper/2203.09645/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [HPatches: A benchmark and evaluation of handcrafted and learned local descriptors](https://mlsanity.com/paper/1704.05939/) (2017)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [Learning Two-View Correspondences and Geometry Using Order-Aware Network](https://mlsanity.com/paper/1908.04964/) (2019)

## Built on by (most cited first, 30 of 126)

- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [RoMa: Robust Dense Feature Matching](https://mlsanity.com/paper/2305.15404/) (2023)
- [Visual Geometry Grounded Deep Structure From Motion](https://mlsanity.com/paper/2312.04563/) (2023)
- [ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer](https://mlsanity.com/paper/2208.14201/) (2022)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)
- [Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed](https://mlsanity.com/paper/2403.04765/) (2024)
- [Detector-Free Structure from Motion](https://mlsanity.com/paper/2306.15669/) (2023)
- [GlueStick: Robust Image Matching by Sticking Points and Lines Together](https://mlsanity.com/paper/2304.02008/) (2023)
- [DeDoDe: Detect, Don't Describe -- Describe, Don't Detect for Local Feature Matching](https://mlsanity.com/paper/2308.08479/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [OmniGlue: Generalizable Feature Matching with Foundation Model Guidance](https://mlsanity.com/paper/2405.12979/) (2024)
- [StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition](https://mlsanity.com/paper/2503.06220/) (2025)
- [MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training](https://mlsanity.com/paper/2501.07556/) (2025)
- [DeDoDe v2: Analyzing and Improving the DeDoDe Keypoint Detector](https://mlsanity.com/paper/2404.08928/) (2024)
- [SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding](https://mlsanity.com/paper/2306.05407/) (2023)
- [PATS: Patch Area Transportation with Subdivision for Local Feature Matching](https://mlsanity.com/paper/2303.07700/) (2023)
- [TopicFM: Robust and Interpretable Topic-Assisted Feature Matching](https://mlsanity.com/paper/2207.00328/) (2022)
- [PMatch: Paired Masked Image Modeling for Dense Geometric Matching](https://mlsanity.com/paper/2303.17342/) (2023)
- [Local Feature Matching Using Deep Learning: A Survey](https://mlsanity.com/paper/2401.17592/) (2024)
- [Adaptive Spot-Guided Transformer for Consistent Local Feature Matching](https://mlsanity.com/paper/2303.16624/) (2023)
- [NOPE-SAC: Neural One-Plane RANSAC for Sparse-View Planar 3D Reconstruction](https://mlsanity.com/paper/2211.16799/) (2022)
- [DeepMatcher: A Deep Transformer-based Network for Robust and Accurate Local Feature Matching](https://mlsanity.com/paper/2301.02993/) (2023)
- [Relative Pose Estimation through Affine Corrections of Monocular Depth Priors](https://mlsanity.com/paper/2501.05446/) (2025)
- [Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints](https://mlsanity.com/paper/2303.02885/) (2023)
- [Segment Anything Model is a Good Teacher for Local Feature Learning](https://mlsanity.com/paper/2309.16992/) (2023)
- [CorresNeRF: Image Correspondence Priors for Neural Radiance Fields](https://mlsanity.com/paper/2312.06642/) (2023)
- [Dense-SfM: Structure from Motion with Dense Consistent Matching](https://mlsanity.com/paper/2501.14277/) (2025)
- [JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba](https://mlsanity.com/paper/2503.03437/) (2025)
- [iComMa: Inverting 3D Gaussian Splatting for Camera Pose Estimation via Comparing and Matching](https://mlsanity.com/paper/2312.09031/) (2023)
- [EarthMatch: Iterative Coregistration for Fine-grained Localization of Astronaut Photography](https://mlsanity.com/paper/2405.05422/) (2024)

## Abstract (arXiv)

Local feature matching is a computationally intensive task at the subpixel level. While detector-based methods coupled with feature descriptors struggle in low-texture scenes, CNN-based methods with a sequential extract-to-match pipeline, fail to make use of the matching capacity of the encoder and tend to overburden the decoder for matching. In contrast, we propose a novel hierarchical extract-and-match transformer, termed as MatchFormer. Inside each stage of the hierarchical encoder, we interleave self-attention for feature extraction and cross-attention for feature matching, yielding a human-intuitive extract-and-match scheme. Such a match-aware encoder releases the overloaded decoder and makes the model highly efficient. Further, combining self- and cross-attention on multi-scale features in a hierarchical architecture improves matching robustness, particularly in low-texture indoor scenes or with less outdoor training data. Thanks to such a strategy, MatchFormer is a multi-win solution in efficiency, robustness, and precision. Compared to the previous best method in indoor pose estimation, our lite MatchFormer has only 45% GFLOPs, yet achieves a +1.3% precision gain and a 41% running speed boost. The large MatchFormer reaches state-of-the-art on four different benchmarks, including indoor pose estimation (ScanNet), outdoor pose estimation (MegaDepth), homography estimation and image matching (HPatch), and visual localization (InLoc).