# The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective arXiv: https://arxiv.org/abs/2108.02238 ยท HTML: https://mlsanity.com/paper/2108.02238/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 116 references that are in the map. - First posted on arXiv 4 Aug 2021. ## Builds on (most cited first, 25 of 25) - [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018) - [Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces](https://mlsanity.com/paper/1903.03545/) (2019) - [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018) - [Deep Learning in Medical Image Registration: A Survey](https://mlsanity.com/paper/1903.02026/) (2019) - [Quicksilver: Fast Predictive Image Registration - a Deep Learning Approach](https://mlsanity.com/paper/1703.10908/) (2017) - [Deep Learning in Medical Image Registration: A Review](https://mlsanity.com/paper/1912.12318/) (2019) - [DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures](https://mlsanity.com/paper/1803.08606/) (2018) - [CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions](https://mlsanity.com/paper/1910.09031/) (2019) - [Label-driven weakly-supervised learning for multimodal deformable image registration](https://mlsanity.com/paper/1711.01666/) (2017) - [Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration](https://mlsanity.com/paper/1911.07042/) (2019) - [Dilated FCN for Multi-Agent 2D/3D Medical Image Registration](https://mlsanity.com/paper/1712.01651/) (2017) - [X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery](https://mlsanity.com/paper/1803.08608/) (2018) - [Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration](https://mlsanity.com/paper/2003.10987/) (2020) - [Multiview 2D/3D Rigid Registration via a Point-Of-Interest Network for Tracking and Triangulation (POINT2)](https://mlsanity.com/paper/1903.03896/) (2019) - [Real-time 2D/3D Registration via CNN Regression](https://mlsanity.com/paper/1507.07505/) (2015) - [UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model](https://mlsanity.com/paper/1709.03450/) (2017) - [Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry](https://mlsanity.com/paper/1805.01026/) (2018) - [Predicting Slice-to-Volume Transformation in Presence of Arbitrary Subject Motion](https://mlsanity.com/paper/1702.08891/) (2017) - [Pose Estimation of Periacetabular Osteotomy Fragments with Intraoperative X-Ray Navigation](https://mlsanity.com/paper/1903.09339/) (2019) - [A Learning-based Method for Online Adjustment of C-arm Cone-Beam CT Source Trajectories for Artifact Avoidance](https://mlsanity.com/paper/2008.06262/) (2020) - [An Interpretable Approach to Automated Severity Scoring in Pelvic Trauma](https://mlsanity.com/paper/2105.10238/) (2021) - [Metric-Driven Learning of Correspondence Weighting for 2-D/3-D Image Registration](https://mlsanity.com/paper/1806.07812/) (2018) - [Localizing dexterous surgical tools in X-ray for image-based navigation](https://mlsanity.com/paper/1901.06672/) (2019) - [Fast and Automatic Periacetabular Osteotomy Fragment Pose Estimation Using Intraoperatively Implanted Fiducials and Single-View Fluoroscopy](https://mlsanity.com/paper/1910.10187/) (2019) - [Deep Learning compatible Differentiable X-ray Projections for Inverse Rendering](https://mlsanity.com/paper/2102.02912/) (2021) ## Built on by (most cited first, 22 of 22) - [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023) - [Local Feature Matching Using Deep Learning: A Survey](https://mlsanity.com/paper/2401.17592/) (2024) - [Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering](https://mlsanity.com/paper/2312.06358/) (2023) - [Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future](https://mlsanity.com/paper/2108.04543/) (2021) - [Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion](https://mlsanity.com/paper/2210.07611/) (2022) - [Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery](https://mlsanity.com/paper/2308.02917/) (2023) - [SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments](https://mlsanity.com/paper/2206.06127/) (2022) - [Embedded Feature Similarity Optimization with Specific Parameter Initialization for 2D/3D Medical Image Registration](https://mlsanity.com/paper/2305.06252/) (2023) - [An Optimization-based Baseline for Rigid 2D/3D Registration Applied to Spine Surgical Navigation Using CMA-ES](https://mlsanity.com/paper/2402.05642/) (2024) - [Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching](https://mlsanity.com/paper/2410.18683/) (2024) - [PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations](https://mlsanity.com/paper/2505.19256/) (2025) - [CNN-based real-time 2D-3D deformable registration from a single X-ray projection](https://mlsanity.com/paper/2212.07692/) (2022) - [Human-AI Collaboration and Explainability for 2D/3D Registration Quality Assurance](https://mlsanity.com/paper/2507.17597/) (2025) - [Intraoperative 2D/3D Registration via Spherical Similarity Learning and Differentiable Levenberg-Marquardt Optimization](https://mlsanity.com/paper/2509.06890/) (2025) - [The Impact of Loss Functions and Scene Representations for 3D/2D Registration on Single-view Fluoroscopic X-ray Pose Estimation](https://mlsanity.com/paper/2308.00214/) (2023) - [A Cranial-Feature-Based Registration Scheme for Robotic Micromanipulation Using a Microscopic Stereo Camera System](https://mlsanity.com/paper/2410.18630/) (2024) - [Rapid patient-specific neural networks for intraoperative X-ray to volume registration](https://mlsanity.com/paper/2503.16309/) (2025) - [Robust and Accurate Multi-view 2D/3D Image Registration with Differentiable X-ray Rendering and Dual Cross-view Constraints](https://mlsanity.com/paper/2506.22191/) (2025) - [NeuralBoneReg: An Instance-Specific Label-Free Point Cloud-Based Method for Multi-Modal Bone Surface Registration](https://mlsanity.com/paper/2511.14286/) (2025) - [Enhanced Landmark Detection Model in Pelvic Fluoroscopy using 2D/3D Registration Loss](https://mlsanity.com/paper/2511.21575/) (2025) - [GeoPose: Patient-agnostic CTA-to-DSA registration through projection-space calibration](https://mlsanity.com/paper/2608.16600/) (2026) - [FleXray: Universal Clinical X-ray Segmentation](https://mlsanity.com/paper/2609.26756/) (2026) ## Abstract (arXiv) Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Further, it is expected that image-based navigation will play a major role in enabling mixed reality environments and autonomous, robotic workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., volumetric imagery or tool models, and 2D images thereof, such as fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness of the optimization objective, hyperparameter selection, and initialization, difficulties around inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.