# Image-and-Spatial Transformer Networks for Structure-Guided Image Registration

arXiv: https://arxiv.org/abs/1907.09200 · HTML: https://mlsanity.com/paper/1907.09200/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2019.

## Built on by (most cited first, 30 of 36)

- [Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning](https://mlsanity.com/paper/2112.04489/) (2021)
- [A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints](https://mlsanity.com/paper/2304.09941/) (2023)
- [Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives](https://mlsanity.com/paper/2308.01265/) (2023)
- [Generative Myocardial Motion Tracking via Latent Space Exploration with Biomechanics-informed Prior](https://mlsanity.com/paper/2206.03830/) (2022)
- [Cross-Modality High-Frequency Transformer for MR Image Super-Resolution](https://mlsanity.com/paper/2203.15314/) (2022)
- [Enhancing Medical Image Registration via Appearance Adjustment Networks](https://mlsanity.com/paper/2103.05213/) (2021)
- [Embedding Gradient-based Optimization in Image Registration Networks](https://mlsanity.com/paper/2112.03915/) (2021)
- [Brain Tumor Sequence Registration with Non-iterative Coarse-to-fine Networks and Dual Deep Supervision](https://mlsanity.com/paper/2211.07876/) (2022)
- [Semantic similarity metrics for learned image registration](https://mlsanity.com/paper/2104.10051/) (2021)
- [Cross-Modality Image Registration using a Training-Time Privileged Third Modality](https://mlsanity.com/paper/2207.12901/) (2022)
- [Learning Accurate Template Matching with Differentiable Coarse-to-Fine Correspondence Refinement](https://mlsanity.com/paper/2303.08438/) (2023)
- [Anatomy-guided Multimodal Registration by Learning Segmentation without Ground Truth: Application to Intraprocedural CBCT/MR Liver Segmentation and Registration](https://mlsanity.com/paper/2104.07056/) (2021)
- [One shot PACS: Patient specific Anatomic Context and Shape prior aware recurrent registration-segmentation of longitudinal thoracic cone beam CTs](https://mlsanity.com/paper/2201.11000/) (2022)
- [One-Shot Medical Landmark Localization by Edge-Guided Transform and Noisy Landmark Refinement](https://mlsanity.com/paper/2208.00453/) (2022)
- [Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network](https://mlsanity.com/paper/2207.10506/) (2022)
- [CAS-Net: Conditional Atlas Generation and Brain Segmentation for Fetal MRI](https://mlsanity.com/paper/2205.08239/) (2022)
- [AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration](https://mlsanity.com/paper/2309.05271/) (2023)
- [Uncertainty quantification in non-rigid image registration via stochastic gradient Markov chain Monte Carlo](https://mlsanity.com/paper/2110.13289/) (2021)
- [Regularized directional representations for medical image registration](https://mlsanity.com/paper/2111.15509/) (2021)
- [Rink-Agnostic Hockey Rink Registration](https://mlsanity.com/paper/2401.01003/) (2023)
- [AtlasMorph: Learning conditional deformable templates for brain MRI](https://mlsanity.com/paper/2511.13609/) (2025)
- [CAR-Net: Unsupervised Co-Attention Guided Registration Network for Joint Registration and Structure Learning](https://mlsanity.com/paper/2106.06637/) (2021)
- [Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation](https://mlsanity.com/paper/2305.06978/) (2023)
- [BigReg: An Efficient Registration Pipeline for High-Resolution X-Ray and Light-Sheet Fluorescence Microscopy](https://mlsanity.com/paper/2404.14807/) (2024)
- [Nuclei-Location Based Point Set Registration of Multi-Stained Whole Slide Images](https://mlsanity.com/paper/2404.17041/) (2024)
- [SAMIR, an efficient registration framework via robust feature learning from SAM](https://mlsanity.com/paper/2509.13629/) (2025)
- [Robust End-to-End Focal Liver Lesion Detection using Unregistered Multiphase Computed Tomography Images](https://mlsanity.com/paper/2112.01535/) (2021)
- [Spatial regularisation for improved accuracy and interpretability in keypoint-based registration](https://mlsanity.com/paper/2503.04499/) (2025)
- [Auto-regressive transformation for image alignment](https://mlsanity.com/paper/2505.04864/) (2025)
- [Read Pointer Meters in complex environments based on a Human-like Alignment and Recognition Algorithm](https://mlsanity.com/paper/2302.14323/) (2023)

## Abstract (arXiv)

Image registration with deep neural networks has become an active field of research and exciting avenue for a long standing problem in medical imaging. The goal is to learn a complex function that maps the appearance of input image pairs to parameters of a spatial transformation in order to align corresponding anatomical structures. We argue and show that the current direct, non-iterative approaches are sub-optimal, in particular if we seek accurate alignment of Structures-of-Interest (SoI). Information about SoI is often available at training time, for example, in form of segmentations or landmarks. We introduce a novel, generic framework, Image-and-Spatial Transformer Networks (ISTNs), to leverage SoI information allowing us to learn new image representations that are optimised for the downstream registration task. Thanks to these representations we can employ a test-specific, iterative refinement over the transformation parameters which yields highly accurate registration even with very limited training data. Performance is demonstrated on pairwise 3D brain registration and illustrative synthetic data.