# CoMIR: Contrastive Multimodal Image Representation for Registration

arXiv: https://arxiv.org/abs/2006.06325 · HTML: https://mlsanity.com/paper/2006.06325/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 30 of 45)

- [Multi-level Feature Learning for Contrastive Multi-view Clustering](https://mlsanity.com/paper/2106.11193/) (2021)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Self-Supervised Equivariant Learning for Oriented Keypoint Detection](https://mlsanity.com/paper/2204.08613/) (2022)
- [Learning Rotation-Equivariant Features for Visual Correspondence](https://mlsanity.com/paper/2303.15472/) (2023)
- [Steerers: A framework for rotation equivariant keypoint descriptors](https://mlsanity.com/paper/2312.02152/) (2023)
- [Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration](https://mlsanity.com/paper/2402.18933/) (2024)
- [Unlocking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imaging](https://mlsanity.com/paper/2308.05764/) (2023)
- [To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance](https://mlsanity.com/paper/2511.12121/) (2025)
- [Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation](https://mlsanity.com/paper/2204.13656/) (2022)
- [Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study](https://mlsanity.com/paper/2103.16262/) (2021)
- [Semantic similarity metrics for learned image registration](https://mlsanity.com/paper/2104.10051/) (2021)
- [ContraReg: Contrastive Learning of Multi-modality Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2206.13434/) (2022)
- [A Deep-Discrete Learning Framework for Spherical Surface Registration](https://mlsanity.com/paper/2203.12999/) (2022)
- [MAD: Modality Agnostic Distance Measure for Image Registration](https://mlsanity.com/paper/2309.02875/) (2023)
- [Knowledge-Augmented Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop](https://mlsanity.com/paper/2104.04968/) (2021)
- [Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment](https://mlsanity.com/paper/2106.14699/) (2021)
- [A Contrastive Distillation Approach for Incremental Semantic Segmentation in Aerial Images](https://mlsanity.com/paper/2112.03814/) (2021)
- [Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations](https://mlsanity.com/paper/2203.00187/) (2022)
- [Deformation equivariant cross-modality image synthesis with paired non-aligned training data](https://mlsanity.com/paper/2208.12491/) (2022)
- [Learning Cross-Scale Visual Representations for Real-Time Image Geo-Localization](https://mlsanity.com/paper/2109.04087/) (2021)
- [Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness](https://mlsanity.com/paper/2205.00001/) (2022)
- [Infusing Lattice Symmetry Priors in Attention Mechanisms for Sample-Efficient Abstract Geometric Reasoning](https://mlsanity.com/paper/2306.03175/) (2023)
- [Cross-Modality Sub-Image Retrieval using Contrastive Multimodal Image Representations](https://mlsanity.com/paper/2201.03597/) (2022)
- [L1BSR: Exploiting Detector Overlap for Self-Supervised Single-Image Super-Resolution of Sentinel-2 L1B Imagery](https://mlsanity.com/paper/2304.06871/) (2023)
- [To Match or Not to Match: Revisiting Image Matching for Reliable Visual Place Recognition](https://mlsanity.com/paper/2504.06116/) (2025)
- [InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals](https://mlsanity.com/paper/2504.09707/) (2025)
- [Can representation learning for multimodal image registration be improved by supervision of intermediate layers?](https://mlsanity.com/paper/2303.00403/) (2023)

## Abstract (arXiv)

We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrastive estimation (InfoNCE). Unlike other contrastive coding methods, used for, e.g., classification, our approach generates image-like representations that contain the information shared between modalities. We introduce a novel, hyperparameter-free modification to InfoNCE, to enforce rotational equivariance of the learnt representations, a property essential to the registration task. We assess the extent of achieved rotational equivariance and the stability of the representations with respect to weight initialization, training set, and hyperparameter settings, on a remote sensing dataset of RGB and near-infrared images. We evaluate the learnt representations through registration of a biomedical dataset of bright-field and second-harmonic generation microscopy images; two modalities with very little apparent correlation. The proposed approach based on CoMIRs significantly outperforms registration of representations created by GAN-based image-to-image translation, as well as a state-of-the-art, application-specific method which takes additional knowledge about the data into account. Code is available at: https://github.com/MIDA-group/CoMIR.