# Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration

arXiv: https://arxiv.org/abs/2402.18933 · HTML: https://mlsanity.com/paper/2402.18933/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Large Deformation Diffeomorphic Image Registration with Laplacian Pyramid Networks](https://mlsanity.com/paper/2006.16148/) (2020)
- [Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning](https://mlsanity.com/paper/2112.04489/) (2021)
- [Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural Networks](https://mlsanity.com/paper/2003.09514/) (2020)
- [Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs](https://mlsanity.com/paper/2003.00152/) (2020)
- [HyperMorph: Amortized Hyperparameter Learning for Image Registration](https://mlsanity.com/paper/2101.01035/) (2021)
- [SAM: Self-supervised Learning of Pixel-wise Anatomical Embeddings in Radiological Images](https://mlsanity.com/paper/2012.02383/) (2020)
- [CoMIR: Contrastive Multimodal Image Representation for Registration](https://mlsanity.com/paper/2006.06325/) (2020)
- [Fast 3D registration with accurate optimisation and little learning for Learn2Reg 2021](https://mlsanity.com/paper/2112.03053/) (2021)
- [Affine Medical Image Registration with Coarse-to-Fine Vision Transformer](https://mlsanity.com/paper/2203.15216/) (2022)
- [Multi-Site Infant Brain Segmentation Algorithms: The iSeg-2019 Challenge](https://mlsanity.com/paper/2007.02096/) (2020)
- [Biomedical image analysis competitions: The state of current participation practice](https://mlsanity.com/paper/2212.08568/) (2022)
- [Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond](https://mlsanity.com/paper/2004.14557/) (2020)
- [Evaluation of MRI to ultrasound registration methods for brain shift correction: The CuRIOUS2018 Challenge](https://mlsanity.com/paper/1904.10535/) (2019)
- [Unsupervised Deformable Image Registration with Absent Correspondences in Pre-operative and Post-Recurrence Brain Tumor MRI Scans](https://mlsanity.com/paper/2206.03900/) (2022)
- [SAMConvex: Fast Discrete Optimization for CT Registration using Self-supervised Anatomical Embedding and Correlation Pyramid](https://mlsanity.com/paper/2307.09727/) (2023)
- [Robust Image Registration with Absent Correspondences in Pre-operative and Follow-up Brain MRI Scans of Diffuse Glioma Patients](https://mlsanity.com/paper/2210.11045/) (2022)
- [Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions](https://mlsanity.com/paper/2203.06810/) (2022)
- [Unsupervised 3D registration through optimization-guided cyclical self-training](https://mlsanity.com/paper/2306.16997/) (2023)
- [JSSR: A Joint Synthesis, Segmentation, and Registration System for 3D Multi-Modal Image Alignment of Large-scale Pathological CT Scans](https://mlsanity.com/paper/2005.12209/) (2020)
- [Matching in the Wild: Learning Anatomical Embeddings for Multi-Modality Images](https://mlsanity.com/paper/2307.03535/) (2023)

## Built on by (most cited first, 17 of 17)

- [uniGradICON: A Foundation Model for Medical Image Registration](https://mlsanity.com/paper/2403.05780/) (2024)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [multiGradICON: A Foundation Model for Multimodal Medical Image Registration](https://mlsanity.com/paper/2408.00221/) (2024)
- [Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework](https://mlsanity.com/paper/2410.10663/) (2024)
- [CABLD: Contrast-Agnostic Brain Landmark Detection with Consistency-Based Regularization](https://mlsanity.com/paper/2411.17845/) (2024)
- [UniReg: Conditional Unified Model for Medical Image Registration](https://mlsanity.com/paper/2503.12868/) (2025)
- [Mono-Modalizing Extremely Heterogeneous Multi-Modal Medical Image Registration](https://mlsanity.com/paper/2506.15596/) (2025)
- [Intraoperative 2D/3D Registration via Spherical Similarity Learning and Differentiable Levenberg-Marquardt Optimization](https://mlsanity.com/paper/2509.06890/) (2025)
- [Uncertainty Estimation for Pretrained Medical Image Registration Models via Transformation Equivariance](https://mlsanity.com/paper/2509.23355/) (2025)
- [Multimodal Diffeomorphic Registration with Neural ODEs and Structural Descriptors](https://mlsanity.com/paper/2512.22689/) (2025)
- [Adaptive Conditional Contrast-Agnostic Deformable Image Registration with Uncertainty Estimation](https://mlsanity.com/paper/2601.05981/) (2026)
- [PCReg-Net: Progressive Contrast-Guided Registration for Cross-Domain Image Alignment](https://mlsanity.com/paper/2602.13304/) (2026)
- [Robust Self-Supervised Cross-Modal Super-Resolution against Real-World Misaligned Observations](https://mlsanity.com/paper/2602.18822/) (2026)
- [CoRe: Joint Optimization with Contrastive Learning for Medical Image Registration](https://mlsanity.com/paper/2603.23694/) (2026)
- [Adapting Frozen Mono-modal Backbones for Multi-modal Registration via Contrast-Agnostic Instance Optimization](https://mlsanity.com/paper/2603.26393/) (2026)
- [Patient-Agnostic Synthetic Pretraining for Efficient Patient-Specific Intraoperative 2D/3D Registration](https://mlsanity.com/paper/2607.23343/) (2026)
- [Benchmarking Intra-Patient 3D Deformable Multimodal Image Registration](https://mlsanity.com/paper/2609.15669/) (2026)

## Abstract (arXiv)

Establishing dense anatomical correspondence across distinct imaging modalities is a foundational yet challenging procedure for numerous medical image analysis studies and image-guided radiotherapy. Existing multi-modality image registration algorithms rely on statistical-based similarity measures or local structural image representations. However, the former is sensitive to locally varying noise, while the latter is not discriminative enough to cope with complex anatomical structures in multimodal scans, causing ambiguity in determining the anatomical correspondence across scans with different modalities. In this paper, we propose a modality-agnostic structural representation learning method, which leverages Deep Neighbourhood Self-similarity (DNS) and anatomy-aware contrastive learning to learn discriminative and contrast-invariance deep structural image representations (DSIR) without the need for anatomical delineations or pre-aligned training images. We evaluate our method on multiphase CT, abdomen MR-CT, and brain MR T1w-T2w registration. Comprehensive results demonstrate that our method is superior to the conventional local structural representation and statistical-based similarity measures in terms of discriminability and accuracy.