mlsanityData to 14 Oct 2026

Understanding Self-Distillation in the Presence of Label Noise

arXiv 2301.13304 ↗First posted 30 Jan 2023cs.LG cs.AI stat.MLcited by 21

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Self-distillation (SD) is the process of first training a enquoteteacher model and then using its predictions to train a enquotestudent model with the textitsame architecture. Specifically, the student's objective function is big(xi*ℓ(teacher's predictions, student's predictions) + (1-xi)*ℓ(given labels, student's predictions)big), where ℓ is some loss function and xi is some parameter ∈ [0,1]. Empirically, SD has been observed to provide performance gains in several settings. In this paper, we theoretically characterize the effect of SD in two supervised learning problems with textitnoisy labels. We first analyze SD for regularized linear regression and show that in the high label noise regime, the optimal value of xi that minimizes the expected error in estimating the ground truth parameter is surprisingly greater than 1. Empirically, we show that xi > 1 works better than xi ≤ 1 even with the cross-entropy loss for several classification datasets when 50% or 30% of the labels are corrupted. Further, we quantify when optimal SD is better than optimal regularization. Next, we analyze SD in the case of logistic regression for binary classification with random label corruption and quantify the range of label corruption in which the student outperforms the teacher in terms of accuracy. To our knowledge, this is the first result of its kind for the cross-entropy loss.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 31 of its 41 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

31 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Robust Active Distillation3 Oct
  2. 2022Revisiting Self-Distillation17 Jun
  3. 2022Knowledge Distillation: Bad Models Can Be Good Role Models28 Mar
  4. 2021Does Knowledge Distillation Really Work?10 Jun
  5. 2021Knowledge distillation: A good teacher is patient and consistent9 Jun
  6. 2021Towards Understanding Knowledge Distillation27 May
  7. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective1 Feb

Other references · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  9. 2018Born Again Neural Networkscited by 546
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2020Meta Pseudo Labelscited by 374
  12. 2019Variational Information Distillation for Knowledge Transfercited by 324
12 more
  1. 2015Unifying distillation and privileged informationcited by 233
  2. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  3. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  4. 2020Does label smoothing mitigate label noise?cited by 204
  5. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  6. 2017Learning from Noisy Labels with Distillationcited by 161
  7. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  8. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 66
  9. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  10. 2019Distillation ≈ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Networkcited by 28
  11. 2022Toward Understanding Privileged Features Distillation in Learning-to-Rankcited by 16
  12. 2020Knowledge Distillation Beyond Model Compressioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 9; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  2. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  3. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  4. 2024Understanding the Gains from Repeated Self-Distillationcited by 15
  5. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 12
  6. 2024Distilling the Knowledge in Data Pruningcited by 11
  7. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  8. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  9. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  10. 2024Retraining with Predicted Hard Labels Provably Increases Model Accuracycited by 5
  11. 2024Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentationcited by 5
  12. 2025Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotationcited by 5
9 more
  1. 2024GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noisecited by 2
  2. 2025The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Modelcited by 2
  3. 2025Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillationcited by 1
  4. 2025Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passingcited by 1
  5. 2026Can we Trust Unreliable Voxels? Exploring 3D Semantic Occupancy Prediction under Label Noisecited by 1
  6. 2024Selfish Evolution: Making Discoveries in Extreme Label Noise with the Help of Overfitting Dynamicscited by 0
  7. 2025Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillationcited by 0
  8. 2026Optimal Self-Distillation for Rectified Flow via Linear Probingcited by 0
  9. 2026Prediction-Only Distillation in Linear and Logistic Regressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Self-discipline on multiple channels
  2. 2024Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels
  3. 2023Random Teachers are Good Teachers
  4. 2025Distillation Scaling Laws
  5. 2026Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting
  6. 2024A Theoretical Analysis of Soft-Label vs Hard-Label Training in Neural Networks
  7. 2026Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
  8. 2023Student as an Inherent Denoiser of Noisy Teacher