# An Artificial Agent for Robust Image Registration

arXiv: https://arxiv.org/abs/1611.10336 · HTML: https://mlsanity.com/paper/1611.10336/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2016.

## Built on by (most cited first, 23 of 23)

- [Deep reinforcement learning in medical imaging: A literature review](https://mlsanity.com/paper/2103.05115/) (2021)
- [Deep Implicit Statistical Shape Models for 3D Medical Image Delineation](https://mlsanity.com/paper/2104.02847/) (2021)
- [Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future](https://mlsanity.com/paper/2108.04543/) (2021)
- [Test-Time Training for Deformable Multi-Scale Image Registration](https://mlsanity.com/paper/2103.13578/) (2021)
- [Reinforcement Learning in Medical Image Analysis: Concepts, Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2206.14302/) (2022)
- [LAPNet: Non-rigid Registration derived in k-space for Magnetic Resonance Imaging](https://mlsanity.com/paper/2107.09060/) (2021)
- [SE(3)-Equivariant and Noise-Invariant 3D Rigid Motion Tracking in Brain MRI](https://mlsanity.com/paper/2312.13534/) (2023)
- [Deep Learning for Medical Image Registration: A Comprehensive Review](https://mlsanity.com/paper/2204.11341/) (2022)
- [Stochastic Planner-Actor-Critic for Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2112.07415/) (2021)
- [InstantGroup: Instant Template Generation for Scalable Group of Brain MRI Registration](https://mlsanity.com/paper/2211.05622/) (2022)
- [Multi-Target Landmark Detection with Incomplete Images via Reinforcement Learning and Shape Prior](https://mlsanity.com/paper/2301.05392/) (2023)
- [Stroke Extraction of Chinese Character Based on Deep Structure Deformable Image Registration](https://mlsanity.com/paper/2307.04341/) (2023)
- [Domain adaptation strategies for 3D reconstruction of the lumbar spine using real fluoroscopy data](https://mlsanity.com/paper/2401.16027/) (2024)
- [Precise Aerial Image Matching based on Deep Homography Estimation](https://mlsanity.com/paper/2107.08768/) (2021)
- [Unsupervised Image Registration Towards Enhancing Performance and Explainability in Cardiac And Brain Image Analysis](https://mlsanity.com/paper/2203.03638/) (2022)
- [Impact of PCA-based preprocessing and different CNN structures on deformable registration of sonograms](https://mlsanity.com/paper/2301.08802/) (2023)
- [DynaWeightPnP: Toward global real-time 3D-2D solver in PnP without correspondences](https://mlsanity.com/paper/2409.18457/) (2024)
- [D-Net: Siamese based Network with Mutual Attention for Volume Alignment](https://mlsanity.com/paper/2101.10248/) (2021)
- [PRANet: Point Cloud Registration with an Artificial Agent](https://mlsanity.com/paper/2109.11349/) (2021)
- [DPCN++: Differentiable Phase Correlation Network for Versatile Pose Registration](https://mlsanity.com/paper/2206.05707/) (2022)
- [A Two-Step Deep Learning Method for 3DCT-2DUS Kidney Registration During Breathing](https://mlsanity.com/paper/2305.13855/) (2023)
- [MorphSeek: Fine-grained Latent Representation-Level Policy Optimization for Deformable Image Registration](https://mlsanity.com/paper/2511.17392/) (2025)
- [LDRNet: Large Deformation Registration Model for Chest CT Registration](https://mlsanity.com/paper/2602.01812/) (2026)

## Abstract (arXiv)

3-D image registration, which involves aligning two or more images, is a critical step in a variety of medical applications from diagnosis to therapy. Image registration is commonly performed by optimizing an image matching metric as a cost function. However, this task is challenging due to the non-convex nature of the matching metric over the plausible registration parameter space and insufficient approaches for a robust optimization. As a result, current approaches are often customized to a specific problem and sensitive to image quality and artifacts. In this paper, we propose a completely different approach to image registration, inspired by how experts perform the task. We first cast the image registration problem as a "strategy learning" process, where the goal is to find the best sequence of motion actions (e.g. up, down, etc.) that yields image alignment. Within this approach, an artificial agent is learned, modeled using deep convolutional neural networks, with 3D raw image data as the input, and the next optimal action as the output. To cope with the dimensionality of the problem, we propose a greedy supervised approach for an end-to-end training, coupled with attention-driven hierarchical strategy. The resulting registration approach inherently encodes both a data-driven matching metric and an optimal registration strategy (policy). We demonstrate, on two 3-D/3-D medical image registration examples with drastically different nature of challenges, that the artificial agent outperforms several state-of-art registration methods by a large margin in terms of both accuracy and robustness.