# Paced-Curriculum Distillation with Prediction and Label Uncertainty for Image Segmentation

arXiv: https://arxiv.org/abs/2302.01049 · HTML: https://mlsanity.com/paper/2302.01049/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 19 of 19)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation](https://mlsanity.com/paper/1707.03718/) (2017)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples](https://mlsanity.com/paper/1704.07433/) (2017)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [When Do Curricula Work?](https://mlsanity.com/paper/2012.03107/) (2020)
- [Dynamic Curriculum Learning for Imbalanced Data Classification](https://mlsanity.com/paper/1901.06783/) (2019)
- [Automatic Curriculum Learning through Value Disagreement](https://mlsanity.com/paper/2006.09641/) (2020)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)
- [Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification](https://mlsanity.com/paper/2009.13698/) (2020)
- [Follow Your Path: a Progressive Method for Knowledge Distillation](https://mlsanity.com/paper/2107.09305/) (2021)
- [Knowledge Distillation via Instance-level Sequence Learning](https://mlsanity.com/paper/2106.10885/) (2021)

## Built on by (most cited first, 6 of 6)

- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation](https://mlsanity.com/paper/2306.03511/) (2023)
- [Advancing Brain Imaging Analysis Step-by-step via Progressive Self-paced Learning](https://mlsanity.com/paper/2407.16128/) (2024)
- [OBSeg: Accurate and Fast Instance Segmentation Framework Using Segmentation Foundation Models with Oriented Bounding Box Prompts](https://mlsanity.com/paper/2401.08174/) (2024)
- [Progressive Learning with Anatomical Priors for Reliable Left Atrial Scar Segmentation from Late Gadolinium Enhancement MRI](https://mlsanity.com/paper/2603.26186/) (2026)
- [UASPL: Uncertainty-Aware Self-Paced Learning with Evidential Neural Networks](https://mlsanity.com/paper/2607.06638/) (2026)

## Abstract (arXiv)

Purpose: In curriculum learning, the idea is to train on easier samples first and gradually increase the difficulty, while in self-paced learning, a pacing function defines the speed to adapt the training progress. While both methods heavily rely on the ability to score the difficulty of data samples, an optimal scoring function is still under exploration. Methodology: Distillation is a knowledge transfer approach where a teacher network guides a student network by feeding a sequence of random samples. We argue that guiding student networks with an efficient curriculum strategy can improve model generalization and robustness. For this purpose, we design an uncertainty-based paced curriculum learning in self distillation for medical image segmentation. We fuse the prediction uncertainty and annotation boundary uncertainty to develop a novel paced-curriculum distillation (PCD). We utilize the teacher model to obtain prediction uncertainty and spatially varying label smoothing with Gaussian kernel to generate segmentation boundary uncertainty from the annotation. We also investigate the robustness of our method by applying various types and severity of image perturbation and corruption. Results: The proposed technique is validated on two medical datasets of breast ultrasound image segmentation and robotassisted surgical scene segmentation and achieved significantly better performance in terms of segmentation and robustness. Conclusion: P-CD improves the performance and obtains better generalization and robustness over the dataset shift. While curriculum learning requires extensive tuning of hyper-parameters for pacing function, the level of performance improvement suppresses this limitation.