# Every Annotation Counts: Multi-label Deep Supervision for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2104.13243 · HTML: https://mlsanity.com/paper/2104.13243/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 76 references that are in the map.
- First posted on arXiv 27 Apr 2021.

## Builds on (most cited first, 30 of 50)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Semi-Supervised Semantic Segmentation with Cross-Consistency Training](https://mlsanity.com/paper/2003.09005/) (2020)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1503.01640/) (2015)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation](https://mlsanity.com/paper/1603.06098/) (2016)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [Adversarial Learning for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/1802.07934/) (2018)

## Built on by (most cited first, 30 of 37)

- [Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer](https://mlsanity.com/paper/2112.04894/) (2021)
- [Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation](https://mlsanity.com/paper/2207.14191/) (2022)
- [Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective](https://mlsanity.com/paper/2302.01735/) (2023)
- [Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation](https://mlsanity.com/paper/2112.00735/) (2021)
- [Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels](https://mlsanity.com/paper/2209.13476/) (2022)
- [Directional Connectivity-based Segmentation of Medical Images](https://mlsanity.com/paper/2304.00145/) (2023)
- [FedMix: Mixed Supervised Federated Learning for Medical Image Segmentation](https://mlsanity.com/paper/2205.01840/) (2022)
- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images](https://mlsanity.com/paper/2303.00865/) (2023)
- [When CNN Meet with ViT: Towards Semi-Supervised Learning for Multi-Class Medical Image Semantic Segmentation](https://mlsanity.com/paper/2208.06449/) (2022)
- [Beyond Pixels: Semi-Supervised Semantic Segmentation with a Multi-scale Patch-based Multi-Label Classifier](https://mlsanity.com/paper/2407.04036/) (2024)
- [Exploring Feature Representation Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2111.10989/) (2021)
- [Compete to Win: Enhancing Pseudo Labels for Barely-supervised Medical Image Segmentation](https://mlsanity.com/paper/2304.07519/) (2023)
- [Disease-informed Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.15728/) (2024)
- [RetiFluidNet: A Self-Adaptive and Multi-Attention Deep Convolutional Network for Retinal OCT Fluid Segmentation](https://mlsanity.com/paper/2209.12468/) (2022)
- [Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention](https://mlsanity.com/paper/2205.04326/) (2022)
- [FCA: Taming Long-tailed Federated Medical Image Classification by Classifier Anchoring](https://mlsanity.com/paper/2305.00738/) (2023)
- [ASLseg: Adapting SAM in the Loop for Semi-supervised Liver Tumor Segmentation](https://mlsanity.com/paper/2312.07969/) (2023)
- [Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Models](https://mlsanity.com/paper/2404.13239/) (2024)
- [Self-Supervised Guided Segmentation Framework for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2209.12440/) (2022)
- [Full-scale Representation Guided Network for Retinal Vessel Segmentation](https://mlsanity.com/paper/2501.18921/) (2025)
- [Is Visual in-Context Learning for Compositional Medical Tasks within Reach?](https://mlsanity.com/paper/2507.00868/) (2025)
- [Soft-CP: A Credible and Effective Data Augmentation for Semantic Segmentation of Medical Lesions](https://mlsanity.com/paper/2203.10507/) (2022)
- [Exemplar Learning for Medical Image Segmentation](https://mlsanity.com/paper/2204.01713/) (2022)
- [Domain Translation via Latent Space Mapping](https://mlsanity.com/paper/2212.03361/) (2022)
- [MetaSeg: Content-Aware Meta-Net for Omni-Supervised Semantic Segmentation](https://mlsanity.com/paper/2401.11738/) (2024)
- [Evaluating Zero-cost Active Learning for Object Detection](https://mlsanity.com/paper/2212.04211/) (2022)
- [From Static to Interactive: Adapting Visual in-Context Learners for User-Driven Tasks](https://mlsanity.com/paper/2604.06748/) (2026)
- [A Joint Framework Towards Class-aware and Class-agnostic Alignment for Few-shot Segmentation](https://mlsanity.com/paper/2211.01310/) (2022)
- [SL: Stable Learning in Source-Free Domain Adaption for Medical Image Segmentation](https://mlsanity.com/paper/2307.12580/) (2023)

## Abstract (arXiv)

Pixel-wise segmentation is one of the most data and annotation hungry tasks in our field. Providing representative and accurate annotations is often mission-critical especially for challenging medical applications. In this paper, we propose a semi-weakly supervised segmentation algorithm to overcome this barrier. Our approach is based on a new formulation of deep supervision and student-teacher model and allows for easy integration of different supervision signals. In contrast to previous work, we show that care has to be taken how deep supervision is integrated in lower layers and we present multi-label deep supervision as the most important secret ingredient for success. With our novel training regime for segmentation that flexibly makes use of images that are either fully labeled, marked with bounding boxes, just global labels, or not at all, we are able to cut the requirement for expensive labels by 94.22% - narrowing the gap to the best fully supervised baseline to only 5% mean IoU. Our approach is validated by extensive experiments on retinal fluid segmentation and we provide an in-depth analysis of the anticipated effect each annotation type can have in boosting segmentation performance.