mlsanityData to 7 Oct 2026

Teacher's pet: understanding and mitigating biases in distillation

arXiv 2106.10494 ↗First posted 19 Jun 2021cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model. Several works have shown that distillation significantly boosts the student's overall performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples. We trace this behaviour to errors made by the teacher distribution being transferred to and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 62 of its 96 references that are in the map. First posted on arXiv 19 Jun 2021.

Builds on

62 of 96 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  11. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  12. 2019When Does Label Smoothing Help?cited by 985
38 more
  1. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  2. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  4. 2020Long-tail learning via logit adjustmentcited by 605
  5. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  6. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  7. 2018Born Again Neural Networkscited by 539
  8. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2019Agnostic Federated Learningcited by 451
  11. 2019BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognitioncited by 403
  12. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 397
  13. 2019On the Efficacy of Knowledge Distillationcited by 377
  14. 2020What is being transferred in transfer learning?cited by 365
  15. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  16. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  17. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  18. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  19. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  20. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  21. 2015Unifying distillation and privileged informationcited by 218
  22. 2020No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problemscited by 208
  23. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  24. 2015Cost Sensitive Learning of Deep Feature Representations from Imbalanced Datacited by 167
  25. 2020Seesaw Loss for Long-Tailed Instance Segmentationcited by 146
  26. 2018Large scale distributed neural network training through online distillationcited by 144
  27. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  28. 2020Feature Space Augmentation for Long-Tailed Datacited by 136
  29. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  30. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  31. 2017The Devil is in the Tails: Fine-grained Classification in the Wildcited by 89
  32. 2019Fairness risk measurescited by 85
  33. 2020Understanding and Improving Knowledge Distillationcited by 83
  34. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  35. 2018The Price of Fair PCA: One Extra Dimensioncited by 67
  36. 2019Leveraging Labeled and Unlabeled Data for Consistent Fair Binary Classificationcited by 60
  37. 2019Adjusting Decision Boundary for Class Imbalanced Learningcited by 52
  38. 2020Selective Classification Can Magnify Disparities Across Groupscited by 35

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 5; 2025: 2; 2026: 3.

20 papers.

Sort:

  1. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  2. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 26
  3. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  4. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  5. 2023Using Early Readouts to Mediate Featural Bias in Distillationcited by 7
  6. 2024DRoP: Distributionally Robust Data Pruningcited by 7
  7. 2022Robust Distillation for Worst-class Performancecited by 6
  8. 2023Debiased Distillation by Transplanting the Last Layercited by 6
  9. 2023What do larger image classifiers memorise?cited by 6
  10. 2024Adaptive Group Robust Ensemble Knowledge Distillationcited by 5
  11. 2024What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Biascited by 4
  12. 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Datacited by 4
8 more
  1. 2024An investigation of structures responsible for gender bias in BERT and DistilBERTcited by 3
  2. 2022Improved Group Robustness via Classifier Retraining on Independent Splitscited by 2
  3. 2023Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillationcited by 2
  4. 2024Disparate Impact on Group Accuracy of Linearization for Private Inferencecited by 2
  5. 2025Does Weak-to-strong Generalization Happen under Spurious Correlations?cited by 1
  6. 2026Knowledge Distillation Must Account for What It Losescited by 1
  7. 2026Anti-Shortcut Distillation via Temporal Negative Knowledge Transfercited by 1
  8. 2026Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Yearcited by 0