mlsanityData to 14 Oct 2026

Deep Probabilistic Supervision for Image Classification

arXiv 2512.24162 ↗First posted 30 Dec 2025cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Supervised training of deep neural networks for classification typically relies on hard targets, which promote overconfidence and can limit calibration, generalization, and robustness. Self-distillation methods aim to mitigate this by leveraging inter-class and sample-specific information present in the model's own predictions, but often remain dependent on hard targets without explicitly modeling predictive uncertainty. With this in mind, we propose Deep Probabilistic Supervision (DPS), a principled learning framework constructing sample-specific target distributions via statistical inference on the model's own predictions, remaining independent of hard targets after initialization. We show that DPS consistently yields higher test accuracy (e.g., +2.0% for DenseNet-264 on ImageNet) and significantly lower Expected Calibration Error (ECE) (-40% ResNet-50, CIFAR-100) than existing self-distillation methods. When combined with a contrastive loss, DPS achieves state-of-the-art robustness under label noise.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 38 of its 55 references that are in the map. First posted on arXiv 30 Dec 2025.

Builds on

38 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Distillation from the Last Mini-Batch for Consistency Regularization30 Mar

Other references · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2017On Calibration of Modern Neural Networkscited by 4,960
  11. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  12. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
25 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  5. 2017Random Erasing Data Augmentationcited by 1,239
  6. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  7. 2019When Does Label Smoothing Help?cited by 988
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  9. 2018Born Again Neural Networkscited by 546
  10. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  12. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  13. 2018Joint Optimization Framework for Learning with Noisy Labelscited by 293
  14. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
  15. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  16. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  17. 2020Confidence-Aware Learning for Deep Neural Networkscited by 113
  18. 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 107
  19. 2018Snapshot Distillation: Teacher-Student Optimization in One Generationcited by 99
  20. 2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
  21. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  22. 2022AdaFocal: Calibration-aware Adaptive Focal Losscited by 32
  23. 2023CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labelscited by 17
  24. 2023ACLS: Adaptive and Conditional Label Smoothing for Network Calibrationcited by 15
  25. 2023ChiMera: Learning with noisy labels by contrasting mixed-up augmentationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Probabilistic Self-supervised Learning via Scoring Rules Minimization
  2. 2024Self-supervised cross-modality learning for uncertainty-aware object detection and recognition in applications which lack pre-labelled training data
  3. 2021Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
  4. 2021Self Supervision to Distillation for Long-Tailed Visual Recognition
  5. 2023Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data
  6. 2025Beyond Accuracy: What Matters in Designing Well-Behaved Image Classification Models?
  7. 2025Dataset Distillation for Pre-Trained Self-Supervised Vision Models
  8. 2022Distillation from heterogeneous unlabeled collections