# Deep Iterative 2D/3D Registration

arXiv: https://arxiv.org/abs/2107.10004 · HTML: https://mlsanity.com/paper/2107.10004/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation](https://mlsanity.com/paper/1805.09707/) (2018)
- [Dilated FCN for Multi-Agent 2D/3D Medical Image Registration](https://mlsanity.com/paper/1712.01651/) (2017)
- [Multiview 2D/3D Rigid Registration via a Point-Of-Interest Network for Tracking and Triangulation (POINT2)](https://mlsanity.com/paper/1903.03896/) (2019)
- [Real-time 2D/3D Registration via CNN Regression](https://mlsanity.com/paper/1507.07505/) (2015)
- [Pose-dependent weights and Domain Randomization for fully automatic X-ray to CT Registration](https://mlsanity.com/paper/2011.07294/) (2020)
- [Learning the Update Operator for 2D/3D Image Registration](https://mlsanity.com/paper/2102.02861/) (2021)

## Built on by (most cited first, 6 of 6)

- [Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering](https://mlsanity.com/paper/2312.06358/) (2023)
- [Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion](https://mlsanity.com/paper/2210.07611/) (2022)
- [Embedded Feature Similarity Optimization with Specific Parameter Initialization for 2D/3D Medical Image Registration](https://mlsanity.com/paper/2305.06252/) (2023)
- [Automatic Multi-View X-Ray/CT Registration Using Bone Substructure Contours](https://mlsanity.com/paper/2506.13292/) (2025)
- [Intraoperative 2D/3D Registration via Spherical Similarity Learning and Differentiable Levenberg-Marquardt Optimization](https://mlsanity.com/paper/2509.06890/) (2025)
- [Patient-Agnostic Synthetic Pretraining for Efficient Patient-Specific Intraoperative 2D/3D Registration](https://mlsanity.com/paper/2607.23343/) (2026)

## Abstract (arXiv)

Deep Learning-based 2D/3D registration methods are highly robust but often lack the necessary registration accuracy for clinical application. A refinement step using the classical optimization-based 2D/3D registration method applied in combination with Deep Learning-based techniques can provide the required accuracy. However, it also increases the runtime. In this work, we propose a novel Deep Learning driven 2D/3D registration framework that can be used end-to-end for iterative registration tasks without relying on any further refinement step. We accomplish this by learning the update step of the 2D/3D registration framework using Point-to-Plane Correspondences. The update step is learned using iterative residual refinement-based optical flow estimation, in combination with the Point-to-Plane correspondence solver embedded as a known operator. Our proposed method achieves an average runtime of around 8s, a mean re-projection distance error of 0.60 ± 0.40 mm with a success ratio of 97 percent and a capture range of 60 mm. The combination of high registration accuracy, high robustness, and fast runtime makes our solution ideal for clinical applications.