mlsanityData to 7 Oct 2026

Better Supervisory Signals by Observing Learning Paths

arXiv 2203.02485 ↗First posted 4 Mar 2022stat.ML cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Better-supervised models might have better performance. In this paper, we first clarify what makes for good supervision for a classification problem, and then explain two existing label refining methods, label smoothing and knowledge distillation, in terms of our proposed criterion. To further answer why and how better supervision emerges, we observe the learning path, i.e., the trajectory of the model's predictions during training, for each training sample. We find that the model can spontaneously refine "bad" labels through a "zig-zag" learning path, which occurs on both toy and real datasets. Observing the learning path not only provides a new perspective for understanding knowledge distillation, overfitting, and learning dynamics, but also reveals that the supervisory signal of a teacher network can be very unstable near the best points in training on real tasks. Inspired by this, we propose a new knowledge distillation scheme, Filter-KD, which improves downstream classification performance in various settings.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

24 of 35 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2019When Does Label Smoothing Help?cited by 985
  12. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
12 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  2. 2018Born Again Neural Networkscited by 539
  3. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  4. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  5. 2019On the Efficacy of Knowledge Distillationcited by 377
  6. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  7. 2019Human uncertainty makes classification more robustcited by 231
  8. 2020Self-Adaptive Training: beyond Empirical Risk Minimizationcited by 126
  9. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  10. 2020Understanding and Improving Knowledge Distillationcited by 83
  11. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  12. 2020S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 8; 2025: 1; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Learning Dynamics of LLM Finetuningcited by 93
  2. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  3. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  4. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  5. 2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 21
  6. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  7. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  8. 2022Smoothing Matters: Momentum Transformer for Domain Adaptive Semantic Segmentationcited by 15
  9. 2023Differentially Private Neural Tangent Kernels for Privacy-Preserving Data Generationcited by 15
  10. 2023Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddingscited by 15
  11. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  12. 2024How to Train the Teacher Model for Effective Knowledge Distillationcited by 13
9 more
  1. 2024lpNTK: Better Generalisation with Less Data via Sample Interaction During Learningcited by 6
  2. 2023Learning Trajectories are Generalization Indicatorscited by 5
  3. 2024Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamicscited by 5
  4. 2023AdaFlood: Adaptive Flood Regularizationcited by 2
  5. 2025Who Gets Credit or Blame? Attributing Accountability in Modern AI Systemscited by 2
  6. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  7. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  8. 2026Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learningcited by 0
  9. 2026Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learningcited by 0