# Joint Registration and Segmentation via Multi-Task Learning for Adaptive Radiotherapy of Prostate Cancer

arXiv: https://arxiv.org/abs/2105.01844 · HTML: https://mlsanity.com/paper/2105.01844/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning](https://mlsanity.com/paper/1711.01239/) (2017)
- [Deep Learning in Medical Image Registration: A Review](https://mlsanity.com/paper/1912.12318/) (2019)
- [DeepAtlas: Joint Semi-Supervised Learning of Image Registration and Segmentation](https://mlsanity.com/paper/1904.08465/) (2019)
- [3D Whole Brain Segmentation using Spatially Localized Atlas Network Tiles](https://mlsanity.com/paper/1903.12152/) (2019)
- [Label-driven weakly-supervised learning for multimodal deformable image registration](https://mlsanity.com/paper/1711.01666/) (2017)
- [Latent Multi-task Architecture Learning](https://mlsanity.com/paper/1705.08142/) (2017)
- [BIRNet: Brain Image Registration Using Dual-Supervised Fully Convolutional Networks](https://mlsanity.com/paper/1802.04692/) (2018)
- [Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity](https://mlsanity.com/paper/1804.10735/) (2018)
- [3D Convolutional Neural Networks Image Registration Based on Efficient Supervised Learning from Artificial Deformations](https://mlsanity.com/paper/1908.10235/) (2019)
- [Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back](https://mlsanity.com/paper/1803.04062/) (2018)
- [A Cross-Stitch Architecture for Joint Registration and Segmentation in Adaptive Radiotherapy](https://mlsanity.com/paper/2004.08122/) (2020)
- [Adversarial optimization for joint registration and segmentation in prostate CT radiotherapy](https://mlsanity.com/paper/1906.12223/) (2019)
- [JSSR: A Joint Synthesis, Segmentation, and Registration System for 3D Multi-Modal Image Alignment of Large-scale Pathological CT Scans](https://mlsanity.com/paper/2005.12209/) (2020)
- [Patient-Specific Finetuning of Deep Learning Models for Adaptive Radiotherapy in Prostate CT](https://mlsanity.com/paper/2002.06927/) (2020)
- [Esophageal Tumor Segmentation in CT Images using Dilated Dense Attention Unet (DDAUnet)](https://mlsanity.com/paper/2012.03242/) (2020)

## Built on by (most cited first, 8 of 8)

- [AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration](https://mlsanity.com/paper/2309.05271/) (2023)
- [VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis](https://mlsanity.com/paper/2410.08397/) (2024)
- [The use of deep learning in interventional radiotherapy (brachytherapy): a review with a focus on open source and open data](https://mlsanity.com/paper/2205.07516/) (2022)
- [Learning Semi-Supervised Medical Image Segmentation from Spatial Registration](https://mlsanity.com/paper/2409.10422/) (2024)
- [Dual-Attention Frequency Fusion at Multi-Scale for Joint Segmentation and Deformable Medical Image Registration](https://mlsanity.com/paper/2409.19658/) (2024)
- [SAM-Guided Robust Representation Learning for One-Shot 3D Medical Image Segmentation](https://mlsanity.com/paper/2504.20501/) (2025)
- [CMRINet: Joint Groupwise Registration and Segmentation for Cardiac Function Quantification from Cine-MRI](https://mlsanity.com/paper/2505.16452/) (2025)
- [Segmentation Regularized Training for Multi-Domain Deep Learning Registration applied to MR-Guided Prostate Cancer Radiotherapy](https://mlsanity.com/paper/2507.06966/) (2025)

## Abstract (arXiv)

Medical image registration and segmentation are two of the most frequent tasks in medical image analysis. As these tasks are complementary and correlated, it would be beneficial to apply them simultaneously in a joint manner. In this paper, we formulate registration and segmentation as a joint problem via a Multi-Task Learning (MTL) setting, allowing these tasks to leverage their strengths and mitigate their weaknesses through the sharing of beneficial information. We propose to merge these tasks not only on the loss level, but on the architectural level as well. We studied this approach in the context of adaptive image-guided radiotherapy for prostate cancer, where planning and follow-up CT images as well as their corresponding contours are available for training. The study involves two datasets from different manufacturers and institutes. The first dataset was divided into training (12 patients) and validation (6 patients), and was used to optimize and validate the methodology, while the second dataset (14 patients) was used as an independent test set. We carried out an extensive quantitative comparison between the quality of the automatically generated contours from different network architectures as well as loss weighting methods. Moreover, we evaluated the quality of the generated deformation vector field (DVF). We show that MTL algorithms outperform their Single-Task Learning (STL) counterparts and achieve better generalization on the independent test set. The best algorithm achieved a mean surface distance of 1.06 ± 0.3 mm, 1.27 ± 0.4 mm, 0.91 ± 0.4 mm, and 1.76 ± 0.8 mm on the validation set for the prostate, seminal vesicles, bladder, and rectum, respectively. The high accuracy of the proposed method combined with the fast inference speed, makes it a promising method for automatic re-contouring of follow-up scans for adaptive radiotherapy.