mlsanityData to 14 Oct 2026

Distilling Balanced Knowledge from a Biased Teacher

arXiv 2506.18496 ↗First posted 23 Jun 2025cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Conventional knowledge distillation, designed for model compression, fails on long-tailed distributions because the teacher model tends to be biased toward head classes and provides limited supervision for tail classes. We propose Long-Tailed Knowledge Distillation (LTKD), a novel framework that reformulates the conventional objective into two components: a cross-group loss, capturing mismatches in prediction distributions across class groups (head, medium, and tail), and a within-group loss, capturing discrepancies within each group's distribution. This decomposition reveals the specific sources of the teacher's bias. To mitigate the inherited bias, LTKD introduces (1) a rebalanced cross-group loss that calibrates the teacher's group-level predictions and (2) a reweighted within-group loss that ensures equal contribution from all groups. Extensive experiments on CIFAR-100-LT, TinyImageNet-LT, and ImageNet-LT demonstrate that LTKD significantly outperforms existing methods in both overall and tail-class accuracy, thereby showing its ability to distill balanced knowledge from a biased teacher for real-world applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 45 of its 53 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

45 of 53 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning23 Nov
  2. 2023FreeKD: Knowledge Distillation via Semantic Frequency Prompt20 Nov
  3. 2023MDCS: More Diverse Experts with Consistency Self-distillation for Long-tailed Recognition19 Aug
  4. 2023Class Attention Transfer Based Knowledge Distillation25 Apr
  5. 2022Knowledge Distillation from A Stronger Teacher21 May
  6. 2022Decoupled Knowledge Distillation16 Mar
  7. 2021Balanced Knowledge Distillation for Long-tailed Learning21 Apr
  8. 2021Distilling Knowledge via Knowledge Review19 Apr
  9. 2021Distilling Virtual Examples for Long-tailed Recognition28 Mar

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2016Wide Residual Networkscited by 3,070
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  10. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
24 more
  1. 2019Relational Knowledge Distillationcited by 889
  2. 2019Contrastive Representation Distillationcited by 748
  3. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  4. 2017Deep Mutual Learningcited by 628
  5. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  7. 2021Deep Long-Tailed Learning: A Surveycited by 417
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  9. 2020Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classificationcited by 175
  10. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 118
  11. 2022Long-Tailed Recognition via Weight Balancingcited by 107
  12. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  13. 2021Self Supervision to Distillation for Long-Tailed Visual Recognitioncited by 73
  14. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  15. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  16. 2023Knowledge Diffusion for Distillationcited by 53
  17. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 52
  18. 2023Global and Local Mixture Consistency Cumulative Learning for Long-tailed Visual Recognitionscited by 43
  19. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 43
  20. 2021Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognitioncited by 27
  21. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 26
  22. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  23. 2021Label-Aware Distribution Calibration for Long-tailed Classificationcited by 13
  24. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios
  2. 2025Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods
  3. 2021Knowledge Distillation as Semiparametric Inference
  4. 2025Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias
  5. 2024Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias
  6. 2023On student-teacher deviations in distillation: does it pay to disobey?
  7. 2021Teacher's pet: understanding and mitigating biases in distillation
  8. 2023Deep Collective Knowledge Distillation