# A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints

arXiv: https://arxiv.org/abs/2304.09941 · HTML: https://mlsanity.com/paper/2304.09941/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 74 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [VoxelMorph: A Learning Framework for Deformable Medical Image Registration](https://mlsanity.com/paper/1809.05231/) (2018)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [LIFT: Learned Invariant Feature Transform](https://mlsanity.com/paper/1603.09114/) (2016)
- [Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces](https://mlsanity.com/paper/1903.03545/) (2019)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration](https://mlsanity.com/paper/1809.06130/) (2018)
- [LF-Net: Learning Local Features from Images](https://mlsanity.com/paper/1805.09662/) (2018)
- [SynthMorph: learning contrast-invariant registration without acquired images](https://mlsanity.com/paper/2004.10282/) (2020)
- [Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters](https://mlsanity.com/paper/1904.00889/) (2019)
- [Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration](https://mlsanity.com/paper/1805.04605/) (2018)
- [Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach](https://mlsanity.com/paper/1703.10908/) (2017)
- [Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration](https://mlsanity.com/paper/1807.03361/) (2018)
- [HyperMorph: Amortized Hyperparameter Learning for Image Registration](https://mlsanity.com/paper/2101.01035/) (2021)
- [End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network](https://mlsanity.com/paper/1704.06065/) (2017)
- [Learning a Probabilistic Model for Diffeomorphic Registration](https://mlsanity.com/paper/1812.07460/) (2018)
- [Conditional Deformable Image Registration with Convolutional Neural Network](https://mlsanity.com/paper/2106.12673/) (2021)
- [TILDE: A Temporally Invariant Learned DEtector](https://mlsanity.com/paper/1411.4568/) (2014)
- [Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations](https://mlsanity.com/paper/1903.09331/) (2019)
- [Image-and-Spatial Transformer Networks for Structure-Guided Image Registration](https://mlsanity.com/paper/1907.09200/) (2019)
- [A Learning Strategy for Contrast-agnostic MRI Segmentation](https://mlsanity.com/paper/2003.01995/) (2020)
- [Label-driven weakly-supervised learning for multimodal deformable image registration](https://mlsanity.com/paper/1711.01666/) (2017)
- [BIRNet: Brain Image Registration Using Dual-Supervised Fully Convolutional Networks](https://mlsanity.com/paper/1802.04692/) (2018)
- [Learning Covariant Feature Detectors](https://mlsanity.com/paper/1605.01224/) (2016)
- [SAME: Deformable Image Registration based on Self-supervised Anatomical Embeddings](https://mlsanity.com/paper/2109.11572/) (2021)
- [Equivariant Filters for Efficient Tracking in 3D Imaging](https://mlsanity.com/paper/2103.10255/) (2021)

## Built on by (most cited first, 25 of 25)

- [SE(3)-Equivariant and Noise-Invariant 3D Rigid Motion Tracking in Brain MRI](https://mlsanity.com/paper/2312.13534/) (2023)
- [MultiMorph: On-demand Atlas Construction](https://mlsanity.com/paper/2504.00247/) (2025)
- [Registration by Regression (RbR): a framework for interpretable and flexible atlas registration](https://mlsanity.com/paper/2404.16781/) (2024)
- [Deep Implicit Optimization enables Robust Learnable Features for Deformable Image Registration](https://mlsanity.com/paper/2406.07361/) (2024)
- [Hierarchical Uncertainty Estimation for Learning-based Registration in Neuroimaging](https://mlsanity.com/paper/2410.09299/) (2024)
- [Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching](https://mlsanity.com/paper/2410.18683/) (2024)
- [Unsupervised Deformable Image Registration with Structural Nonparametric Smoothing](https://mlsanity.com/paper/2506.10813/) (2025)
- [Encoder-Only Image Registration](https://mlsanity.com/paper/2509.00451/) (2025)
- [BigReg: An Efficient Registration Pipeline for High-Resolution X-Ray and Light-Sheet Fluorescence Microscopy](https://mlsanity.com/paper/2404.14807/) (2024)
- [CABLD: Contrast-Agnostic Brain Landmark Detection with Consistency-Based Regularization](https://mlsanity.com/paper/2411.17845/) (2024)
- [CARL: A Framework for Equivariant Image Registration](https://mlsanity.com/paper/2405.16738/) (2024)
- [A Novel Streamline-based diffusion MRI Tractography Registration Method with Probabilistic Keypoint Detection](https://mlsanity.com/paper/2503.02481/) (2025)
- [Spatial regularisation for improved accuracy and interpretability in keypoint-based registration](https://mlsanity.com/paper/2503.04499/) (2025)
- [Deep Biomechanically-Guided Interpolation for Keypoint-Based Brain Shift Registration](https://mlsanity.com/paper/2508.13762/) (2025)
- [Polaffini: A feature-based approach for robust affine and polyaffine image registration](https://mlsanity.com/paper/2602.17337/) (2026)
- [A Unified Deep Learning Framework for Motion Correction in Medical Imaging](https://mlsanity.com/paper/2409.14204/) (2024)
- [RealKeyMorph: Keypoints in Real-world Coordinates for Resolution-agnostic Image Registration](https://mlsanity.com/paper/2506.10344/) (2025)
- [DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapy](https://mlsanity.com/paper/2508.10260/) (2025)
- [K-Stain: Keypoint-Driven Correspondence for H&E-to-IHC Virtual Staining](https://mlsanity.com/paper/2511.06709/) (2025)
- [Equivariant symmetry-aware head pose estimation for fetal MRI](https://mlsanity.com/paper/2512.04890/) (2025)
- [Multimodal Diffeomorphic Registration with Neural ODEs and Structural Descriptors](https://mlsanity.com/paper/2512.22689/) (2025)
- [Data-driven registration and modeling of brain deformation for image-guided neurosurgery](https://mlsanity.com/paper/2602.10155/) (2026)
- [TractoRC: A Unified Probabilistic Learning Framework for Joint Tractography Registration and Clustering](https://mlsanity.com/paper/2603.10418/) (2026)
- [CT-Guided Spatially-varying Regularization for Voxel-Wise Deformable Whole-Body PET Registration](https://mlsanity.com/paper/2604.22905/) (2026)
- [Prediction of Rectal Cancer Regrowth from Longitudinal Endoscopy](https://mlsanity.com/paper/2605.12855/) (2026)

## Abstract (arXiv)

We present KeyMorph, a deep learning-based image registration framework that relies on automatically detecting corresponding keypoints. State-of-the-art deep learning methods for registration often are not robust to large misalignments, are not interpretable, and do not incorporate the symmetries of the problem. In addition, most models produce only a single prediction at test-time. Our core insight which addresses these shortcomings is that corresponding keypoints between images can be used to obtain the optimal transformation via a differentiable closed-form expression. We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints. This framework not only leads to substantially more robust registration but also yields better interpretability, since the keypoints reveal which parts of the image are driving the final alignment. Moreover, KeyMorph can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering. Finally, we show how multiple deformation fields can be computed efficiently and in closed-form at test time corresponding to different transformation variants. We demonstrate the proposed framework in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements. Our code is available at https://github.com/alanqrwang/keymorph.