# A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration

arXiv: https://arxiv.org/abs/2507.18551 · HTML: https://mlsanity.com/paper/2507.18551/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 26 of 26)

- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation](https://mlsanity.com/paper/2304.03608/) (2023)
- [TILDE: A Temporally Invariant Learned DEtector](https://mlsanity.com/paper/1411.4568/) (2014)
- [XoFTR: Cross-modal Feature Matching Transformer](https://mlsanity.com/paper/2404.09692/) (2024)
- [MINIMA: Modality Invariant Image Matching](https://mlsanity.com/paper/2412.19412/) (2024)
- [MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training](https://mlsanity.com/paper/2501.07556/) (2025)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Semi-Supervised Keypoint Detector and Descriptor for Retinal Image Matching](https://mlsanity.com/paper/2207.07932/) (2022)
- [Estimation of Large Motion in Lung CT by Integrating Regularized Keypoint Correspondences into Dense Deformable Registration](https://mlsanity.com/paper/1807.00467/) (2018)
- [Slice-to-volume medical image registration: a survey](https://mlsanity.com/paper/1702.01636/) (2017)
- [Global Multi-modal 2D/3D Registration via Local Descriptors Learning](https://mlsanity.com/paper/2205.03439/) (2022)
- [Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations](https://mlsanity.com/paper/2309.08747/) (2023)
- [Joint detection and matching of feature points in multimodal images](https://mlsanity.com/paper/1810.12941/) (2018)
- [BrainMorph: A Foundational Keypoint Model for Robust and Flexible Brain MRI Registration](https://mlsanity.com/paper/2405.14019/) (2024)
- [Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts](https://mlsanity.com/paper/2410.19378/) (2024)
- [Patient-Specific Real-Time Segmentation in Trackerless Brain Ultrasound](https://mlsanity.com/paper/2405.09959/) (2024)
- [A Feature-Driven Active Framework for Ultrasound-Based Brain Shift Compensation](https://mlsanity.com/paper/1803.07682/) (2018)
- [Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models](https://mlsanity.com/paper/2506.02419/) (2025)
- [Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasound](https://mlsanity.com/paper/2409.08169/) (2024)
- [Automatic Landmarks Correspondence Detection in Medical Images with an Application to Deformable Image Registration](https://mlsanity.com/paper/2109.02722/) (2021)
- [MedNet: Pre-trained Convolutional Neural Network Model for the Medical Imaging Tasks](https://mlsanity.com/paper/2110.06512/) (2021)
- [Driving Points Prediction For Abdominal Probabilistic Registration](https://mlsanity.com/paper/2208.03232/) (2022)
- [An End-to-end Deep Learning Approach for Landmark Detection and Matching in Medical Images](https://mlsanity.com/paper/2001.07434/) (2020)
- [Intraoperative Registration by Cross-Modal Inverse Neural Rendering](https://mlsanity.com/paper/2409.11983/) (2024)

## Built on by (most cited first, 3 of 3)

- [A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgery](https://mlsanity.com/paper/2606.00630/) (2026)
- [Data-driven registration and modeling of brain deformation for image-guided neurosurgery](https://mlsanity.com/paper/2602.10155/) (2026)
- [What neurosurgeons need to see: synthetic intra-operative MRI from ultrasound for brain-shift compensation in brain tumour surgery](https://mlsanity.com/paper/2606.07658/) (2026)

## Abstract (arXiv)

Intraoperative registration of real-time ultrasound (iUS) to preoperative Magnetic Resonance Imaging (MRI) remains an unsolved problem due to severe modality-specific differences in appearance, resolution, and field-of-view. To address this, we propose a novel 3D cross-modal keypoint descriptor for MRI-iUS matching and registration. Our approach employs a patient-specific matching-by-synthesis approach, generating synthetic iUS volumes from preoperative MRI. This enables supervised contrastive training to learn a shared descriptor space. A probabilistic keypoint detection strategy is then employed to identify anatomically salient and modality-consistent locations. During training, a curriculum-based triplet loss with dynamic hard negative mining is used to learn descriptors that are i) robust to iUS artifacts such as speckle noise and limited coverage, and ii) rotation-invariant. At inference, the method detects keypoints in MR and real iUS images and identifies sparse matches, which are then used to perform rigid registration. Our approach is evaluated using 3D MRI-iUS pairs from the ReMIND dataset. Experiments show that our approach outperforms state-of-the-art keypoint matching methods across 11 patients, with an average precision of 69.8%. For image registration, our method achieves a competitive mean Target Registration Error of 2.39 mm on the ReMIND2Reg benchmark. Compared to existing iUS-MR registration approaches, our framework is interpretable, requires no manual initialization, and shows robustness to iUS field-of-view variation. Code, data and model weights are available at https://github.com/morozovdd/CrossKEY.