# Atlas-ISTN: Joint Segmentation, Registration and Atlas Construction with Image-and-Spatial Transformer Networks

arXiv: https://arxiv.org/abs/2012.10533 · HTML: https://mlsanity.com/paper/2012.10533/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Robust machine learning segmentation for large-scale analysis of heterogeneous clinical brain MRI datasets](https://mlsanity.com/paper/2209.02032/) (2022)
- [Vision Transformers in Medical Computer Vision -- A Contemplative Retrospection](https://mlsanity.com/paper/2203.15269/) (2022)
- [Anatomy-aware and acquisition-agnostic joint registration with SynthMorph](https://mlsanity.com/paper/2301.11329/) (2023)
- [Aladdin: Joint Atlas Building and Diffeomorphic Registration Learning with Pairwise Alignment](https://mlsanity.com/paper/2202.03563/) (2022)
- [MultiMorph: On-demand Atlas Construction](https://mlsanity.com/paper/2504.00247/) (2025)
- [Improved 3D Whole Heart Geometry from Sparse CMR Slices](https://mlsanity.com/paper/2408.07532/) (2024)
- [CAS-Net: Conditional Atlas Generation and Brain Segmentation for Fetal MRI](https://mlsanity.com/paper/2205.08239/) (2022)
- [SpaceJAM: a Lightweight and Regularization-free Method for Fast Joint Alignment of Images](https://mlsanity.com/paper/2407.11850/) (2024)
- [Joint segmentation and discontinuity-preserving deformable registration: Application to cardiac cine-MR images](https://mlsanity.com/paper/2211.13828/) (2022)
- [CartiMorph: a framework for automated knee articular cartilage morphometrics](https://mlsanity.com/paper/2308.01981/) (2023)
- [Rink-Agnostic Hockey Rink Registration](https://mlsanity.com/paper/2401.01003/) (2023)
- [CINeMA: Conditional Implicit Neural Multi-Modal Atlas for a Spatio-Temporal Representation of the Perinatal Brain](https://mlsanity.com/paper/2506.09668/) (2025)
- [AtlasMorph: Learning conditional deformable templates for brain MRI](https://mlsanity.com/paper/2511.13609/) (2025)
- [Pay Attention to the Atlas: Atlas-Guided Test-Time Adaptation Method for Robust 3D Medical Image Segmentation](https://mlsanity.com/paper/2307.00676/) (2023)
- [Detecting Hypo-plastic Left Heart Syndrome in Fetal Ultrasound via Disease-specific Atlas Maps](https://mlsanity.com/paper/2107.02643/) (2021)
- [High-Resolution Maps of Left Atrial Displacements and Strains Estimated with 3D Cine MRI using Online Learning Neural Networks](https://mlsanity.com/paper/2312.09387/) (2023)
- [Cardiovascular disease classification using radiomics and geometric features from cardiac CT](https://mlsanity.com/paper/2506.22226/) (2025)
- [FastJAM: a Fast Joint Alignment Model for Images](https://mlsanity.com/paper/2510.22842/) (2025)
- [Ultrasound Image Segmentation of Thyroid Nodule via Latent Semantic Feature Co-Registration](https://mlsanity.com/paper/2310.09221/) (2023)
- [Multi-task learning for joint weakly-supervised segmentation and aortic arch anomaly classification in fetal cardiac MRI](https://mlsanity.com/paper/2311.07234/) (2023)
- [SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation](https://mlsanity.com/paper/2504.20501/) (2025)
- [An Efficient Model-Driven Groupwise Approach for Atlas Construction](https://mlsanity.com/paper/2508.10743/) (2025)
- [INFANiTE: Implicit Neural representation for high-resolution Fetal brain spatio-temporal Atlas learNing from clinical Thick-slicE MRI](https://mlsanity.com/paper/2605.09977/) (2026)
- [GRC-ProbNet: Uncertainty-aware Feature Extraction for Cardiovascular Disease Classification](https://mlsanity.com/paper/2607.10357/) (2026)

## Abstract (arXiv)

Deep learning models for semantic segmentation are able to learn powerful representations for pixel-wise predictions, but are sensitive to noise at test time and do not guarantee a plausible topology. Image registration models on the other hand are able to warp known topologies to target images as a means of segmentation, but typically require large amounts of training data, and have not widely been benchmarked against pixel-wise segmentation models. We propose Atlas-ISTN, a framework that jointly learns segmentation and registration on 2D and 3D image data, and constructs a population-derived atlas in the process. Atlas-ISTN learns to segment multiple structures of interest and to register the constructed, topologically consistent atlas labelmap to an intermediate pixel-wise segmentation. Additionally, Atlas-ISTN allows for test time refinement of the model's parameters to optimize the alignment of the atlas labelmap to an intermediate pixel-wise segmentation. This process both mitigates for noise in the target image that can result in spurious pixel-wise predictions, as well as improves upon the one-pass prediction of the model. Benefits of the Atlas-ISTN framework are demonstrated qualitatively and quantitatively on 2D synthetic data and 3D cardiac computed tomography and brain magnetic resonance image data, out-performing both segmentation and registration baseline models. Atlas-ISTN also provides inter-subject correspondence of the structures of interest, enabling population-level shape and motion analysis.