# XoFTR: Cross-modal Feature Matching Transformer

arXiv: https://arxiv.org/abs/2404.09692 · HTML: https://mlsanity.com/paper/2404.09692/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 92 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [HPatches: A benchmark and evaluation of handcrafted and learned local descriptors](https://mlsanity.com/paper/1704.05939/) (2017)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [DKM: Dense Kernelized Feature Matching for Geometry Estimation](https://mlsanity.com/paper/2202.00667/) (2022)
- [Neighbourhood Consensus Networks](https://mlsanity.com/paper/1810.10510/) (2018)
- [ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer](https://mlsanity.com/paper/2208.14201/) (2022)
- [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Patch2Pix: Epipolar-Guided Pixel-Level Correspondences](https://mlsanity.com/paper/2012.01909/) (2020)

## Built on by (most cited first, 30 of 48)

- [MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training](https://mlsanity.com/paper/2501.07556/) (2025)
- [JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba](https://mlsanity.com/paper/2503.03437/) (2025)
- [ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translation](https://mlsanity.com/paper/2509.24878/) (2025)
- [OrthoLoC: UAV 6-DoF Localization and Calibration Using Orthographic Geodata](https://mlsanity.com/paper/2509.18350/) (2025)
- [Multi-Resolution SAR and Optical Remote Sensing Image Registration Methods: A Review, Datasets, and Future Perspectives](https://mlsanity.com/paper/2502.01002/) (2025)
- [Deep Learning Reforms Image Matching: A Survey and Outlook](https://mlsanity.com/paper/2506.04619/) (2025)
- [MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching](https://mlsanity.com/paper/2501.11299/) (2025)
- [MapGlue: Multimodal Remote Sensing Image Matching](https://mlsanity.com/paper/2503.16185/) (2025)
- [Learning Dense Feature Matching via Lifting Single 2D Image to 3D Space](https://mlsanity.com/paper/2507.00392/) (2025)
- [GDROS: A Geometry-Guided Dense Registration Framework for Optical-SAR Images under Large Geometric Transformations](https://mlsanity.com/paper/2511.00598/) (2025)
- [TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation](https://mlsanity.com/paper/2602.19430/) (2026)
- [A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration](https://mlsanity.com/paper/2507.18551/) (2025)
- [SOMA-1M: A Large-Scale SAR-Optical Multi-resolution Alignment Dataset for Multi-Task Remote Sensing](https://mlsanity.com/paper/2602.05480/) (2026)
- [CRFT: Consistent-Recurrent Feature Flow Transformer for Cross-Modal Image Registration](https://mlsanity.com/paper/2604.05689/) (2026)
- [XPoint: A Self-Supervised Visual-State-Space based Architecture for Multispectral Image Registration](https://mlsanity.com/paper/2411.07430/) (2024)
- [SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for Unpaired RGB+Thermal 3D Reconstruction](https://mlsanity.com/paper/2603.18774/) (2026)
- [SCC-Loc: A Unified Semantic Cascade Consensus Framework for UAV Thermal Geo-Localization](https://mlsanity.com/paper/2604.03120/) (2026)
- [Are Pretrained Image Matchers Good Enough for SAR-Optical Satellite Registration?](https://mlsanity.com/paper/2604.10217/) (2026)
- [PromptMID: Modal Invariant Descriptors Based on Diffusion and Vision Foundation Models for Optical-SAR Image Matching](https://mlsanity.com/paper/2502.18104/) (2025)
- [Grid-Reg: Detector-Free Gridized Feature Learning and Matching for Large-Scale SAR-Optical Image Registration](https://mlsanity.com/paper/2507.04233/) (2025)
- [CM-Bench: A Comprehensive Cross-Modal Feature Matching Benchmark Bridging Visible and Infrared Images](https://mlsanity.com/paper/2603.12690/) (2026)
- [AnyMatch: Supercharging Universal Multi-Modal Image Matching with Large-Scale Single-View Images](https://mlsanity.com/paper/2606.31077/) (2026)
- [REMM:Rotation-Equivariant Framework for End-to-End Multimodal Image Matching](https://mlsanity.com/paper/2407.11637/) (2024)
- [Flow Intelligence: Robust Feature Matching via Temporal Signature Correlation](https://mlsanity.com/paper/2504.11949/) (2025)
- [Semi-distributed Cross-modal Air-Ground Relative Localization](https://mlsanity.com/paper/2511.06749/) (2025)
- [Coarse-to-Fine Non-Rigid Registration for Side-Scan Sonar Mosaicking](https://mlsanity.com/paper/2512.00052/) (2025)
- [OpenNavMap: Multi-Session Appearance-Based Topometric Mapping for Scalable Visual Navigation](https://mlsanity.com/paper/2601.12291/) (2026)
- [Multi-Sensor Matching with HyperNetworks](https://mlsanity.com/paper/2601.12325/) (2026)
- [LST-SLAM: A Stereo Thermal SLAM System for Kilometer-Scale Dynamic Environments](https://mlsanity.com/paper/2602.20925/) (2026)
- [No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency](https://mlsanity.com/paper/2602.23559/) (2026)

## Abstract (arXiv)

We introduce, XoFTR, a cross-modal cross-view method for local feature matching between thermal infrared (TIR) and visible images. Unlike visible images, TIR images are less susceptible to adverse lighting and weather conditions but present difficulties in matching due to significant texture and intensity differences. Current hand-crafted and learning-based methods for visible-TIR matching fall short in handling viewpoint, scale, and texture diversities. To address this, XoFTR incorporates masked image modeling pre-training and fine-tuning with pseudo-thermal image augmentation to handle the modality differences. Additionally, we introduce a refined matching pipeline that adjusts for scale discrepancies and enhances match reliability through sub-pixel level refinement. To validate our approach, we collect a comprehensive visible-thermal dataset, and show that our method outperforms existing methods on many benchmarks.