# MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching

arXiv: https://arxiv.org/abs/2501.11299 · HTML: https://mlsanity.com/paper/2501.11299/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 74 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [Detection and Tracking Meet Drones Challenge](https://mlsanity.com/paper/2001.06303/) (2020)
- [Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion](https://mlsanity.com/paper/2308.12469/) (2023)
- [ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation](https://mlsanity.com/paper/2304.03608/) (2023)
- [The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation](https://mlsanity.com/paper/2306.01923/) (2023)
- [XFeat: Accelerated Features for Lightweight Image Matching](https://mlsanity.com/paper/2404.19174/) (2024)
- [Ambiguous Medical Image Segmentation using Diffusion Models](https://mlsanity.com/paper/2304.04745/) (2023)
- [ROSE: A Retinal OCT-Angiography Vessel Segmentation Dataset and New Model](https://mlsanity.com/paper/2007.05201/) (2020)
- [ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction](https://mlsanity.com/paper/2112.02906/) (2021)
- [GIM: Learning Generalizable Image Matcher From Internet Videos](https://mlsanity.com/paper/2402.11095/) (2024)
- [SiLK -- Simple Learned Keypoints](https://mlsanity.com/paper/2304.06194/) (2023)
- [XoFTR: Cross-modal Feature Matching Transformer](https://mlsanity.com/paper/2404.09692/) (2024)
- [The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion](https://mlsanity.com/paper/1807.01569/) (2018)
- [GLAMpoints: Greedily Learned Accurate Match points](https://mlsanity.com/paper/1908.06812/) (2019)
- [Semi-Supervised Keypoint Detector and Descriptor for Retinal Image Matching](https://mlsanity.com/paper/2207.07932/) (2022)
- [Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity](https://mlsanity.com/paper/2103.16871/) (2021)
- [ReDFeat: Recoupling Detection and Description for Multimodal Feature Learning](https://mlsanity.com/paper/2205.07439/) (2022)
- [Reverse Knowledge Distillation: Training a Large Model using a Small One for Retinal Image Matching on Limited Data](https://mlsanity.com/paper/2307.10698/) (2023)
- [RIFT2: Speeding-up RIFT with A New Rotation-Invariance Technique](https://mlsanity.com/paper/2303.00319/) (2023)
- [Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network](https://mlsanity.com/paper/2207.10506/) (2022)
- [A Novel Deep Learning Pipeline for Retinal Vessel Detection in Fluorescein Angiography](https://mlsanity.com/paper/1907.02946/) (2019)
- [A Detector-oblivious Multi-arm Network for Keypoint Matching](https://mlsanity.com/paper/2104.00947/) (2021)
- [MEMO: Dataset and Methods for Robust Multimodal Retinal Image Registration with Large or Small Vessel Density Differences](https://mlsanity.com/paper/2309.14550/) (2023)

## Built on by (most cited first, 6 of 6)

- [Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification](https://mlsanity.com/paper/2509.23310/) (2025)
- [SURE: Semi-dense Uncertainty-REfined Feature Matching](https://mlsanity.com/paper/2603.04869/) (2026)
- [CM-Bench: A Comprehensive Cross-Modal Feature Matching Benchmark Bridging Visible and Infrared Images](https://mlsanity.com/paper/2603.12690/) (2026)
- [Operational Change Detection for Geographical Information: Overview and Challenges](https://mlsanity.com/paper/2503.14109/) (2025)
- [OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding](https://mlsanity.com/paper/2603.24876/) (2026)
- [Modality-Invariant Coarse-to-Fine Retinal Image Registration](https://mlsanity.com/paper/2608.14829/) (2026)

## Abstract (arXiv)

Many keypoint detection and description methods have been proposed for image matching or registration. While these methods demonstrate promising performance for single-modality image matching, they often struggle with multimodal data because the descriptors trained on single-modality data tend to lack robustness against the non-linear variations present in multimodal data. Extending such methods to multimodal image matching often requires well-aligned multimodal data to learn modality-invariant descriptors. However, acquiring such data is often costly and impractical in many real-world scenarios. To address this challenge, we propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching using only single-modality training data. Specifically, we propose a novel latent feature aggregation module and a cumulative hybrid aggregation module to enhance the base keypoint descriptors trained on single-modality data by leveraging pre-trained features from Stable Diffusion models. %, our approach generates robust and invariant features across diverse and unknown modalities. We validate our method with recent keypoint detection and description methods in three multimodal retinal image datasets (CF-FA, CF-OCT, EMA-OCTA) and two remote sensing datasets (Optical-SAR and Optical-NIR). Extensive experiments demonstrate that the proposed MIFNet is able to learn modality-invariant feature for multimodal image matching without accessing the targeted modality and has good zero-shot generalization ability. The code will be released at https://github.com/lyp-deeplearning/MIFNet.