mlsanityData to 7 Oct 2026

Volumetric medical image segmentation through dual self-distillation in U-shaped networks

arXiv 2306.03271 ↗First posted 5 Jun 2023eess.IV cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

U-shaped networks and its variants have demonstrated exceptional results for medical image segmentation. In this paper, we propose a novel dual self-distillation (DSD) framework in U-shaped networks for volumetric medical image segmentation. DSD distills knowledge from the ground-truth segmentation labels to the decoder layers. Additionally, DSD also distills knowledge from the deepest decoder and encoder layer to the shallower decoder and encoder layers respectively of a single U-shaped network. DSD is a general training strategy that could be attached to the backbone architecture of any U-shaped network to further improve its segmentation performance. We attached DSD on several state-of-the-art U-shaped backbones, and extensive experiments on various public 3D medical image segmentation datasets (cardiac substructure, brain tumor and Hippocampus) demonstrated significant improvement over the same backbones without DSD. On average, after attaching DSD to the U-shaped backbones, we observed an increase of 2.82%, 4.53% and 1.3% in Dice similarity score, a decrease of 7.15 mm, 6.48 mm and 0.76 mm in the Hausdorff distance, for cardiac substructure, brain tumor and Hippocampus segmentation, respectively. These improvements were achieved with negligible increase in the number of trainable parameters and training time. Our proposed DSD framework also led to significant qualitative improvements for cardiac substructure, brain tumor and Hippocampus segmentation over the U-shaped backbones. The source code is publicly available at https://github.com/soumbane/DualSelfDistillation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

35 of 56 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  12. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
23 more
  1. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  2. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
  3. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
  4. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  5. 2021The Medical Segmentation Decathloncited by 620
  6. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 614
  7. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  8. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  9. 2022MONAI: An open-source framework for deep learning in healthcarecited by 438
  10. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  11. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  12. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  13. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  14. 2021TransBTS: Multimodal Brain Tumor Segmentation Using Transformercited by 265
  15. 2021CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentationcited by 227
  16. 2020Medical Image Segmentation Using Deep Learning: A Surveycited by 163
  17. 2016Multivariate mixture model for myocardium segmentation combining multi-source imagescited by 133
  18. 2021Efficient Medical Image Segmentation Based on Knowledge Distillationcited by 43
  19. 2017Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalitiescited by 25
  20. 2021LGD: Label-guided Self-distillation for Object Detectioncited by 19
  21. 2022X-Metric: An N-Dimensional Information-Theoretic Framework for Groupwise Registration and Deep Combined Computingcited by 10
  22. 2022MISSU: 3D Medical Image Segmentation via Self-distilling TransUNetcited by 7
  23. 2017Multi-Planar Deep Segmentation Networks for Cardiac Substructures from MRI and CTcited by 5

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025DRBD-Mamba for Robust and Efficient Brain Tumor Segmentation with Analytical Insightscited by 3
  2. 2025Modality-AGnostic Image Cascade (MAGIC) for Multi-Modality Cardiac Substructure Segmentationcited by 0