# Transfer learning from a sparsely annotated dataset of 3D medical images

arXiv: https://arxiv.org/abs/2311.05032 · HTML: https://mlsanity.com/paper/2311.05032/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning](https://mlsanity.com/paper/1602.03409/) (2016)
- [The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge](https://mlsanity.com/paper/1912.01054/) (2019)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)
- [VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images](https://mlsanity.com/paper/2001.09193/) (2020)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentation](https://mlsanity.com/paper/1904.06346/) (2019)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- [An application of cascaded 3D fully convolutional networks for medical image segmentation](https://mlsanity.com/paper/1803.05431/) (2018)
- [Federated Multi-organ Segmentation with Inconsistent Labels](https://mlsanity.com/paper/2206.07156/) (2022)
- [Cats or CAT scans: transfer learning from natural or medical image source datasets?](https://mlsanity.com/paper/1810.05444/) (2018)
- [PyMIC: A deep learning toolkit for annotation-efficient medical image segmentation](https://mlsanity.com/paper/2208.09350/) (2022)
- [Understanding the Mechanisms of Deep Transfer Learning for Medical Images](https://mlsanity.com/paper/1704.06040/) (2017)
- [Incremental Learning for Multi-organ Segmentation with Partially Labeled Datasets](https://mlsanity.com/paper/2103.04526/) (2021)
- [Healthy versus pathological learning transferability in shoulder muscle MRI segmentation using deep convolutional encoder-decoders](https://mlsanity.com/paper/1901.01620/) (2019)

## Abstract (arXiv)

Transfer learning leverages pre-trained model features from a large dataset to save time and resources when training new models for various tasks, potentially enhancing performance. Due to the lack of large datasets in the medical imaging domain, transfer learning from one medical imaging model to other medical imaging models has not been widely explored. This study explores the use of transfer learning to improve the performance of deep convolutional neural networks for organ segmentation in medical imaging. A base segmentation model (3D U-Net) was trained on a large and sparsely annotated dataset; its weights were used for transfer learning on four new down-stream segmentation tasks for which a fully annotated dataset was available. We analyzed the training set size's influence to simulate scarce data. The results showed that transfer learning from the base model was beneficial when small datasets were available, providing significant performance improvements; where fine-tuning the base model is more beneficial than updating all the network weights with vanilla transfer learning. Transfer learning with fine-tuning increased the performance by up to 0.129 (+28%) Dice score than experiments trained from scratch, and on average 23 experiments increased the performance by 0.029 Dice score in the new segmentation tasks. The study also showed that cross-modality transfer learning using CT scans was beneficial. The findings of this study demonstrate the potential of transfer learning to improve the efficiency of annotation and increase the accessibility of accurate organ segmentation in medical imaging, ultimately leading to improved patient care. We made the network definition and weights publicly available to benefit other users and researchers.