mlsanityData to 14 Oct 2026

Weighted Distillation with Unlabeled Examples

arXiv 2210.06711 ↗First posted 13 Oct 2022cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Distillation with unlabeled examples is a popular and powerful method for training deep neural networks in settings where the amount of labeled data is limited: A large ''teacher'' neural network is trained on the labeled data available, and then it is used to generate labels on an unlabeled dataset (typically much larger in size). These labels are then utilized to train the smaller ''student'' model which will actually be deployed. Naturally, the success of the approach depends on the quality of the teacher's labels, since the student could be confused if trained on inaccurate data. This paper proposes a principled approach for addressing this issue based on a ''debiasing'' reweighting of the student's loss function tailored to the distillation training paradigm. Our method is hyper-parameter free, data-agnostic, and simple to implement. We demonstrate significant improvements on popular academic datasets and we accompany our results with a theoretical analysis which rigorously justifies the performance of our method in certain settings.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 30 of its 69 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

30 of 69 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  10. 2019When Does Label Smoothing Help?cited by 988
  11. 2019Contrastive Representation Distillationcited by 748
  12. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
18 more
  1. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  2. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  3. 2018Born Again Neural Networkscited by 546
  4. 2019Confidence Regularized Self-Trainingcited by 419
  5. 2020Meta Pseudo Labelscited by 374
  6. 2020Rethinking Pre-training and Self-trainingcited by 352
  7. 2019Variational Information Distillation for Knowledge Transfercited by 324
  8. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  9. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  10. 2020Does label smoothing mitigate label noise?cited by 204
  11. 2021Does Knowledge Distillation Really Work?cited by 190
  12. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  13. 2020Wasserstein Contrastive Representation Distillationcited by 64
  14. 2018Stochastic subgradient method converges at the rate O(k-1/4) on weakly convex functionscited by 32
  15. 2020Subclass Distillationcited by 23
  16. 2017Fidelity-Weighted Learningcited by 19
  17. 2021Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyondcited by 16
  18. 2021Constrained Instance and Class Reweighting for Robust Learning under Label Noisecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 0; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  2. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  3. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  4. 2024How to Train the Teacher Model for Effective Knowledge Distillationcited by 13
  5. 2024Revisiting Ensembling in One-Shot Federated Learningcited by 13
  6. 2023SLaM: Student-Label Mixing for Distillation with Unlabeled Examplescited by 10
  7. 2023Towards an On-device Agent for Text Rewritingcited by 7
  8. 2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
  9. 2024Linear Projections of Teacher Embeddings for Few-Class Distillationcited by 1
  10. 2023MyriadAL: Active Few Shot Learning for Histopathologycited by 0
  11. 2023Student as an Inherent Denoiser of Noisy Teachercited by 0
  12. 2026From Skeletons to Pixels: Few-Shot Precise Event Spotting via Representation and Prediction Distillationcited by 0
1 more
  1. 2026Mobility Helps Learning: Unsupervised Model Adaptation for Object Recognition via Movementcited by 0