# Segmentation with mixed supervision: Confidence maximization helps knowledge distillation

arXiv: https://arxiv.org/abs/2109.10902 · HTML: https://mlsanity.com/paper/2109.10902/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 30 of 35)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation](https://mlsanity.com/paper/1907.07034/) (2019)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Simple Does It: Weakly Supervised Instance and Semantic Segmentation](https://mlsanity.com/paper/1603.07485/) (2016)
- [Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation](https://mlsanity.com/paper/1502.02734/) (2015)
- [FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference](https://mlsanity.com/paper/1902.10421/) (2019)
- [DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks](https://mlsanity.com/paper/1605.07866/) (2016)
- [Semi-Supervised Medical Image Segmentation via Learning Consistency under Transformations](https://mlsanity.com/paper/1911.01218/) (2019)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [Constrained-CNN losses for weakly supervised segmentation](https://mlsanity.com/paper/1805.04628/) (2018)
- [Deep Co-Training for Semi-Supervised Image Segmentation](https://mlsanity.com/paper/1903.11233/) (2019)
- [Semi-Supervised Brain Lesion Segmentation with an Adapted Mean Teacher Model](https://mlsanity.com/paper/1903.01248/) (2019)
- [Source-Relaxed Domain Adaptation for Image Segmentation](https://mlsanity.com/paper/2005.03697/) (2020)
- [Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training](https://mlsanity.com/paper/1804.02586/) (2018)
- [Semi-Supervised and Task-Driven Data Augmentation](https://mlsanity.com/paper/1902.05396/) (2019)
- [Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision](https://mlsanity.com/paper/2004.06816/) (2020)
- [Uncertainty guided semi-supervised segmentation of retinal layers in OCT images](https://mlsanity.com/paper/2103.02083/) (2021)
- [Curriculum semi-supervised segmentation](https://mlsanity.com/paper/1904.05236/) (2019)
- [Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/1506.04924/) (2015)
- [Deep semi-supervised segmentation with weight-averaged consistency targets](https://mlsanity.com/paper/1807.04657/) (2018)
- [3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study](https://mlsanity.com/paper/1612.03925/) (2016)
- [Deep Learning with Mixed Supervision for Brain Tumor Segmentation](https://mlsanity.com/paper/1812.04571/) (2018)
- [Self-paced and self-consistent co-training for semi-supervised image segmentation](https://mlsanity.com/paper/2011.00325/) (2020)
- [Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization](https://mlsanity.com/paper/2103.04813/) (2021)

## Built on by (most cited first, 6 of 6)

- [PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation](https://mlsanity.com/paper/2208.05669/) (2022)
- [Beyond Pixel-Wise Supervision for Medical Image Segmentation: From Traditional Models to Foundation Models](https://mlsanity.com/paper/2404.13239/) (2024)
- [A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models](https://mlsanity.com/paper/2502.15959/) (2025)
- [Self and Mixed Supervision to Improve Training Labels for Multi-Class Medical Image Segmentation](https://mlsanity.com/paper/2403.03882/) (2024)
- [Deep Learning Approaches for Medical Imaging Under Varying Degrees of Label Availability: A Comprehensive Survey](https://mlsanity.com/paper/2504.11588/) (2025)
- [From Clinical Intent to Clinical Model: Autonomous Coding-Agents for Clinician-driven AI Development](https://mlsanity.com/paper/2604.17110/) (2026)

## Abstract (arXiv)

Despite achieving promising results in a breadth of medical image segmentation tasks, deep neural networks require large training datasets with pixel-wise annotations. Obtaining these curated datasets is a cumbersome process which limits the applicability in scenarios. Mixed supervision is an appealing alternative for mitigating this obstacle. In this work, we propose a dual-branch architecture, where the upper branch (teacher) receives strong annotations, while the bottom one (student) is driven by limited supervision and guided by the upper branch. Combined with a standard cross-entropy loss over the labeled pixels, our novel formulation integrates two important terms: (i) a Shannon entropy loss defined over the less-supervised images, which encourages confident student predictions in the bottom branch; and (ii) a KL divergence term, which transfers the knowledge (i.e., predictions) of the strongly supervised branch to the less-supervised branch and guides the entropy (student-confidence) term to avoid trivial solutions. We show that the synergy between the entropy and KL divergence yields substantial improvements in performance. We also discuss an interesting link between Shannon-entropy minimization and standard pseudo-mask generation, and argue that the former should be preferred over the latter for leveraging information from unlabeled pixels. We evaluate the effectiveness of the proposed formulation through a series of quantitative and qualitative experiments using two publicly available datasets. Results demonstrate that our method significantly outperforms other strategies for semantic segmentation within a mixed-supervision framework, as well as recent semi-supervised approaches. Our code is publicly available: https://github.com/by-liu/ConfKD.