mlsanityData to 7 Oct 2026

Learning From Biased Soft Labels

arXiv 2302.08155 ↗First posted 16 Feb 2023cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation has been widely adopted in a variety of tasks and has achieved remarkable successes. Since its inception, many researchers have been intrigued by the dark knowledge hidden in the outputs of the teacher model. Recently, a study has demonstrated that knowledge distillation and label smoothing can be unified as learning from soft labels. Consequently, how to measure the effectiveness of the soft labels becomes an important question. Most existing theories have stringent constraints on the teacher model or data distribution, and many assumptions imply that the soft labels are close to the ground-truth labels. This paper studies whether biased soft labels are still effective. We present two more comprehensive indicators to measure the effectiveness of such soft labels. Based on the two indicators, we give sufficient conditions to ensure biased soft label based learners are classifier-consistent and ERM learnable. The theory is applied to three weakly-supervised frameworks. Experimental results validate that biased soft labels can also teach good students, which corroborates the soundness of the theory.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

34 of 69 references.

References · 34

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2016Wide Residual Networkscited by 3,057
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  6. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  7. 2019When Does Label Smoothing Help?cited by 985
  8. 2017Deep Mutual Learningcited by 621
  9. 2018Born Again Neural Networkscited by 539
  10. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  11. 2013Do Deep Nets Really Need to be Deep?cited by 466
  12. 2015Policy Distillationcited by 414
22 more
  1. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  2. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  3. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  4. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  5. 2017Learning from Noisy Labels with Distillationcited by 160
  6. 2021Towards Understanding Knowledge Distillationcited by 158
  7. 2014Label Distribution Learningcited by 122
  8. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  9. 2021Deep Learning with Label Differential Privacycited by 113
  10. 2020Progressive Identification of True Labels for Partial-Label Learningcited by 102
  11. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  12. 2020Provably Consistent Partial-Label Learningcited by 92
  13. 2020Understanding and Improving Knowledge Distillationcited by 83
  14. 2021Antipodes of Label Differential Privacy: PATE and ALIBIcited by 50
  15. 2021Instance-Dependent Partial Label Learningcited by 49
  16. 2021Teachers Do More Than Teach: Compressing Image-to-Image Modelscited by 37
  17. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  18. 2021Generalization bounds via distillationcited by 33
  19. 2020Knowledge Transfer via Dense Cross-Layer Mutual-Distillationcited by 32
  20. 2021Knowledge Distillation as Semiparametric Inferencecited by 29
  21. 2021On the Robustness of Average Losses for Partial-Label Learningcited by 20
  22. 2019Self-supervised Label Augmentation via Input Transformationscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 2; 2026: 3.

8 papers.

  1. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  2. 2023Binary Classification with Confidence Differencecited by 13
  3. 2025U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detectioncited by 12
  4. 2024Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labelscited by 4
  5. 2025Distributions In, Distributions Out: The Case for Soft-Label Trainingcited by 2
  6. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  7. 2026Enhancing Automatic Chord Recognition via Pseudo-Labeling and Knowledge Distillationcited by 0
  8. 2026Recent advances in weakly supervised learning: New supervision paradigms, assumption relaxations, and practical solutionscited by 0