# REMM:Rotation-Equivariant Framework for End-to-End Multimodal Image Matching

arXiv: https://arxiv.org/abs/2407.11637 · HTML: https://mlsanity.com/paper/2407.11637/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [DKM: Dense Kernelized Feature Matching for Geometry Estimation](https://mlsanity.com/paper/2202.00667/) (2022)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [SOSNet: Second Order Similarity Regularization for Local Descriptor Learning](https://mlsanity.com/paper/1904.05019/) (2019)
- [Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters](https://mlsanity.com/paper/1904.00889/) (2019)
- [ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction](https://mlsanity.com/paper/2112.02906/) (2021)
- [XoFTR: Cross-modal Feature Matching Transformer](https://mlsanity.com/paper/2404.09692/) (2024)
- [The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion](https://mlsanity.com/paper/1807.01569/) (2018)
- [Adaptive Assignment for Geometry Aware Local Feature Matching](https://mlsanity.com/paper/2207.08427/) (2022)
- [Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity](https://mlsanity.com/paper/2103.16871/) (2021)
- [ReDFeat: Recoupling Detection and Description for Multimodal Feature Learning](https://mlsanity.com/paper/2205.07439/) (2022)
- [Learning Rotation-Equivariant Features for Visual Correspondence](https://mlsanity.com/paper/2303.15472/) (2023)
- [A Robust Multimodal Remote Sensing Image Registration Method and System Using Steerable Filters with First- and Second-order Gradients](https://mlsanity.com/paper/2202.13347/) (2022)
- [Advances and Challenges in Multimodal Remote Sensing Image Registration](https://mlsanity.com/paper/2302.00912/) (2023)
- [RIFT: Multi-modal Image Matching Based on Radiation-invariant Feature Transform](https://mlsanity.com/paper/1804.09493/) (2018)
- [Nonlinear Intensity, Scale and Rotation Invariant Matching for Multimodal Images](https://mlsanity.com/paper/2302.14239/) (2023)

## Abstract (arXiv)

We present REMM, a rotation-equivariant framework for end-to-end multimodal image matching, which fully encodes rotational differences of descriptors in the whole matching pipeline. Previous learning-based methods mainly focus on extracting modal-invariant descriptors, while consistently ignoring the rotational invariance. In this paper, we demonstrate that our REMM is very useful for multimodal image matching, including multimodal feature learning module and cyclic shift module. We first learn modal-invariant features through the multimodal feature learning module. Then, we design the cyclic shift module to rotationally encode the descriptors, greatly improving the performance of rotation-equivariant matching, which makes them robust to any angle. To validate our method, we establish a comprehensive rotation and scale-matching benchmark for evaluating the anti-rotation performance of multimodal images, which contains a combination of multi-angle and multi-scale transformations from four publicly available datasets. Extensive experiments show that our method outperforms existing methods in benchmarking and generalizes well to independent datasets. Additionally, we conducted an in-depth analysis of the key components of the REMM to validate the improvements brought about by the cyclic shift module. Code and dataset at https://github.com/HanNieWHU/REMM.