# Dilated FCN for Multi-Agent 2D/3D Medical Image Registration

arXiv: https://arxiv.org/abs/1712.01651 · HTML: https://mlsanity.com/paper/1712.01651/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2017.

## Built on by (most cited first, 22 of 22)

- [The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective](https://mlsanity.com/paper/2108.02238/) (2021)
- [Modeling the Probabilistic Distribution of Unlabeled Data forOne-shot Medical Image Segmentation](https://mlsanity.com/paper/2102.02033/) (2021)
- [Test-Time Training for Deformable Multi-Scale Image Registration](https://mlsanity.com/paper/2103.13578/) (2021)
- [Reinforcement Learning in Medical Image Analysis: Concepts, Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2206.14302/) (2022)
- [End-to-end Ultrasound Frame to Volume Registration](https://mlsanity.com/paper/2107.06449/) (2021)
- [Low-Dose CT Denoising via Sinogram Inner-Structure Transformer](https://mlsanity.com/paper/2204.03163/) (2022)
- [Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion](https://mlsanity.com/paper/2210.07611/) (2022)
- [Deep Learning for Medical Image Registration: A Comprehensive Review](https://mlsanity.com/paper/2204.11341/) (2022)
- [Deep Iterative 2D/3D Registration](https://mlsanity.com/paper/2107.10004/) (2021)
- [IGCN: Image-to-graph Convolutional Network for 2D/3D Deformable Registration](https://mlsanity.com/paper/2111.00484/) (2021)
- [An Optimization-based Baseline for Rigid 2D/3D Registration Applied to Spine Surgical Navigation Using CMA-ES](https://mlsanity.com/paper/2402.05642/) (2024)
- [Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies](https://mlsanity.com/paper/2512.01913/) (2025)
- [A Deep Registration Method for Accurate Quantification of Joint Space Narrowing Progression in Rheumatoid Arthritis](https://mlsanity.com/paper/2304.13938/) (2023)
- [Learning the Update Operator for 2D/3D Image Registration](https://mlsanity.com/paper/2102.02861/) (2021)
- [Transfer Learning from an Artificial Radiograph-landmark Dataset for Registration of the Anatomic Skull Model to Dual Fluoroscopic X-ray Images](https://mlsanity.com/paper/2108.06466/) (2021)
- [Domain adaptation strategies for 3D reconstruction of the lumbar spine using real fluoroscopy data](https://mlsanity.com/paper/2401.16027/) (2024)
- [GuidedMorph: Two-Stage Deformable Registration for Breast MRI](https://mlsanity.com/paper/2505.13414/) (2025)
- [Shape-based pose estimation for automatic standard views of the knee](https://mlsanity.com/paper/2305.16717/) (2023)
- [DynaWeightPnP: Toward global real-time 3D-2D solver in PnP without correspondences](https://mlsanity.com/paper/2409.18457/) (2024)
- [LayersReg: A Layer-by-Layer Progressive Regressor for Reliable Intraoperative 3D/2D Registration](https://mlsanity.com/paper/2606.26647/) (2026)
- [A Rigid Registration Method in TEVAR](https://mlsanity.com/paper/2104.14273/) (2021)
- [Masked Registration and Autoencoding of CT Images for Predictive Tibia Reconstruction](https://mlsanity.com/paper/2512.09525/) (2025)

## Abstract (arXiv)

2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.