# Global Multi-modal 2D/3D Registration via Local Descriptors Learning

arXiv: https://arxiv.org/abs/2205.03439 · HTML: https://mlsanity.com/paper/2205.03439/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 6 May 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC](https://mlsanity.com/paper/2002.12324/) (2020)
- [Neighbourhood Consensus Networks](https://mlsanity.com/paper/1810.10510/) (2018)
- [ROBIN: a Graph-Theoretic Approach to Reject Outliers in Robust Estimation using Invariants](https://mlsanity.com/paper/2011.03659/) (2020)
- [Estimation of Large Motion in Lung CT by Integrating Regularized Keypoint Correspondences into Dense Deformable Registration](https://mlsanity.com/paper/1807.00467/) (2018)
- [Evaluation of MRI to ultrasound registration methods for brain shift correction: The CuRIOUS2018 Challenge](https://mlsanity.com/paper/1904.10535/) (2019)
- [Self-Supervised 3D Keypoint Learning for Ego-motion Estimation](https://mlsanity.com/paper/1912.03426/) (2019)
- [SEKD: Self-Evolving Keypoint Detection and Description](https://mlsanity.com/paper/2006.05077/) (2020)

## Built on by (most cited first, 13 of 13)

- [DISA: DIfferentiable Similarity Approximation for Universal Multimodal Registration](https://mlsanity.com/paper/2307.09931/) (2023)
- [Epicardium Prompt-guided Real-time Cardiac Ultrasound Frame-to-volume Registration](https://mlsanity.com/paper/2406.14534/) (2024)
- [TC-DiffRecon: Texture coordination MRI reconstruction method based on diffusion model and modified MF-UNet method](https://mlsanity.com/paper/2402.11274/) (2024)
- [DreamReg: Belief-Driven World Model for 2D-3D Ultrasound Registration](https://mlsanity.com/paper/2606.18825/) (2026)
- [TRUSTED: The Paired 3D Transabdominal Ultrasound and CT Human Data for Kidney Segmentation and Registration Research](https://mlsanity.com/paper/2310.12646/) (2023)
- [A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration](https://mlsanity.com/paper/2507.18551/) (2025)
- [On the Localization of Ultrasound Image Slices within Point Distribution Models](https://mlsanity.com/paper/2309.00372/) (2023)
- [Deep Regression 2D-3D Ultrasound Registration for Liver Motion Correction in Focal Tumor Thermal Ablation](https://mlsanity.com/paper/2410.02579/) (2024)
- [Intraoperative 2D/3D Registration via Spherical Similarity Learning and Differentiable Levenberg-Marquardt Optimization](https://mlsanity.com/paper/2509.06890/) (2025)
- [The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation](https://mlsanity.com/paper/2401.08123/) (2024)
- [Automatic Image Unfolding and Stitching Framework for Esophageal Lining Video Based on Density-Weighted Feature Matching](https://mlsanity.com/paper/2410.01148/) (2024)
- [A Novel Framework for Integrating 3D Ultrasound into Percutaneous Liver Tumour Ablation](https://mlsanity.com/paper/2506.21162/) (2025)
- [Patient-Agnostic Synthetic Pretraining for Efficient Patient-Specific Intraoperative 2D/3D Registration](https://mlsanity.com/paper/2607.23343/) (2026)

## Abstract (arXiv)

Multi-modal registration is a required step for many image-guided procedures, especially ultrasound-guided interventions that require anatomical context. While a number of such registration algorithms are already available, they all require a good initialization to succeed due to the challenging appearance of ultrasound images and the arbitrary coordinate system they are acquired in. In this paper, we present a novel approach to solve the problem of registration of an ultrasound sweep to a pre-operative image. We learn dense keypoint descriptors from which we then estimate the registration. We show that our method overcomes the challenges inherent to registration tasks with freehand ultrasound sweeps, namely, the multi-modality and multidimensionality of the data in addition to lack of precise ground truth and low amounts of training examples. We derive a registration method that is fast, generic, fully automatic, does not require any initialization and can naturally generate visualizations aiding interpretability and explainability. Our approach is evaluated on a clinical dataset of paired MR volumes and ultrasound sequences.