mlsanityData to 14 Oct 2026

Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation

arXiv 2305.05010 ↗First posted 8 May 2023cs.LG cs.CLcited by 25

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation is a popular technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output distribution with the teacher's output distribution. In this work, we argue that such a learning objective is sub-optimal because there exists a discrepancy between the teacher's output distribution and the ground truth label distribution. Therefore, forcing the student to blindly imitate the unreliable teacher output distribution leads to inferior performance. To this end, we propose a novel knowledge distillation objective PTLoss by first representing the vanilla KL-based distillation loss function via a Maclaurin series and then perturbing the leading-order terms in this series. This perturbed loss implicitly transforms the original teacher into a proxy teacher with a distribution closer to the ground truth distribution. We establish the theoretical connection between this "distribution closeness" and the student model generalizability, which enables us to select the PTLoss's perturbation coefficients in a principled way. Extensive experiments on five datasets demonstrate PTLoss can significantly improve the distillation effectiveness for teachers of various scales.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 24 of its 34 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

24 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Decoupled Knowledge Distillation16 Mar
  2. 2022Better Supervisory Signals by Observing Learning Paths4 Mar
  3. 2021Teacher's pet: understanding and mitigating biases in distillation19 Jun
  4. 2021Does Knowledge Distillation Really Work?10 Jun
  5. 2021Distilling Knowledge via Knowledge Review19 Apr
  6. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective1 Feb

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2019When Does Label Smoothing Help?cited by 988
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2017Focal Loss for Dense Object Detectioncited by 752
  11. 2019Contrastive Representation Distillationcited by 748
  12. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
6 more
  1. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 63
  2. 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
  3. 2021Generalization bounds via distillationcited by 34
  4. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  5. 2019Towards Near-imperceptible Steganographic Textcited by 32
  6. 2020Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Codingcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 8; 2026: 6.

25 papers.

Sort:

By area:

  1. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  2. 2024Logit Standardization in Knowledge Distillationcited by 102
  3. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  4. 2024Direct Preference Knowledge Distillation for Large Language Modelscited by 21
  5. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  6. 2024Transferring Backdoors between Large Language Models by Knowledge Distillationcited by 12
  7. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  8. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Modelscited by 8
  9. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architecturescited by 5
  10. 2024FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devicescited by 5
  11. 2025CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizerscited by 4
  12. 2024ProFuser: Progressive Fusion of Large Language Modelscited by 3
13 more
  1. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  2. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  3. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimizationcited by 1
  4. 2025On Membership Inference Attacks in Knowledge Distillationcited by 1
  5. 2025What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry Viewcited by 1
  6. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversitycited by 1
  7. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  8. 2023Local Boosting for Weakly-Supervised Learningcited by 0
  9. 2024Estimating Human Poses Across Datasets: A Unified Skeleton and Multi-Teacher Distillation Approachcited by 0
  10. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  11. 2026REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiencycited by 0
  12. 2026MobileSAM2: Lightweight Segment Anything for Spatial Intelligencecited by 0
  13. 2026Learning from imperfect teachers for low-resource acoustic generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic View
  2. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence
  3. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
  4. 2023Comparative Knowledge Distillation
  5. 2021Knowledge Distillation as Semiparametric Inference
  6. 2023On student-teacher deviations in distillation: does it pay to disobey?
  7. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  8. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation