# Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities

arXiv: https://arxiv.org/abs/1704.03379 · HTML: https://mlsanity.com/paper/1704.03379/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2017.

## Built on by (most cited first, 25 of 25)

- [UniverSeg: Universal Medical Image Segmentation](https://mlsanity.com/paper/2304.06131/) (2023)
- [ImageCAS: A Large-Scale Dataset and Benchmark for Coronary Artery Segmentation based on Computed Tomography Angiography Images](https://mlsanity.com/paper/2211.01607/) (2022)
- [D-Former: A U-shaped Dilated Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2201.00462/) (2022)
- [Comprehensive Multimodal Segmentation in Medical Imaging: Combining YOLOv8 with SAM and HQ-SAM Models](https://mlsanity.com/paper/2310.12995/) (2023)
- [Detect-and-Segment: a Deep Learning Approach to Automate Wound Image Segmentation](https://mlsanity.com/paper/2111.01590/) (2021)
- [Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentation](https://mlsanity.com/paper/2405.18511/) (2024)
- [Multi-task Learning by Leveraging the Semantic Information](https://mlsanity.com/paper/2103.02546/) (2021)
- [Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalities](https://mlsanity.com/paper/2406.11636/) (2024)
- [Implicit U-Net for volumetric medical image segmentation](https://mlsanity.com/paper/2206.15217/) (2022)
- [Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors](https://mlsanity.com/paper/2207.13502/) (2022)
- [Autoregressive Sequence Modeling for 3D Medical Image Representation](https://mlsanity.com/paper/2409.08691/) (2024)
- [Learning from Partially Overlapping Labels: Image Segmentation under Annotation Shift](https://mlsanity.com/paper/2107.05938/) (2021)
- [Towards more precise automatic analysis: a comprehensive survey of deep learning-based multi-organ segmentation](https://mlsanity.com/paper/2303.00232/) (2023)
- [Deep Conditional Shape Models for 3D cardiac image segmentation](https://mlsanity.com/paper/2310.10756/) (2023)
- [Detailed delineation of the fetal brain in diffusion MRI via multi-task learning](https://mlsanity.com/paper/2409.06716/) (2024)
- [Hybrid Dual Mean-Teacher Network With Double-Uncertainty Guidance for Semi-Supervised Segmentation of MRI Scans](https://mlsanity.com/paper/2303.05126/) (2023)
- [Volumetric medical image segmentation through dual self-distillation in U-shaped networks](https://mlsanity.com/paper/2306.03271/) (2023)
- [On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis](https://mlsanity.com/paper/2306.13276/) (2023)
- [A Unified Framework for Joint Detection of Lacunes and Enlarged Perivascular Spaces](https://mlsanity.com/paper/2603.04243/) (2026)
- [TricycleGAN: Unsupervised Image Synthesis and Segmentation Based on Shape Priors](https://mlsanity.com/paper/2102.02690/) (2021)
- [An overview of deep learning in medical imaging](https://mlsanity.com/paper/2202.08546/) (2022)
- [Design Perspectives of Multitask Deep Learning Models and Applications](https://mlsanity.com/paper/2209.13444/) (2022)
- [Automated Diagnosis of Cardiovascular Diseases from Cardiac Magnetic Resonance Imaging Using Deep Learning Models: A Review](https://mlsanity.com/paper/2210.14909/) (2022)
- [Multi-task Learning for Optical Coherence Tomography Angiography (OCTA) Vessel Segmentation](https://mlsanity.com/paper/2311.02266/) (2023)
- [MT-Depth: Multi-task Instance feature analysis for the Depth Completion](https://mlsanity.com/paper/2512.04734/) (2025)

## Abstract (arXiv)

Automatic segmentation of medical images is an important task for many clinical applications. In practice, a wide range of anatomical structures are visualised using different imaging modalities. In this paper, we investigate whether a single convolutional neural network (CNN) can be trained to perform different segmentation tasks. A single CNN is trained to segment six tissues in MR brain images, the pectoral muscle in MR breast images, and the coronary arteries in cardiac CTA. The CNN therefore learns to identify the imaging modality, the visualised anatomical structures, and the tissue classes. For each of the three tasks (brain MRI, breast MRI and cardiac CTA), this combined training procedure resulted in a segmentation performance equivalent to that of a CNN trained specifically for that task, demonstrating the high capacity of CNN architectures. Hence, a single system could be used in clinical practice to automatically perform diverse segmentation tasks without task-specific training.