mlsanityData to 7 Oct 2026

Parameter-Efficient and Student-Friendly Knowledge Distillation

arXiv 2205.15308 ↗First posted 28 May 2022cs.LG cs.AI cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) has been extensively employed to transfer the knowledge from a large teacher model to the smaller students, where the parameters of the teacher are fixed (or partially) during training. Recent studies show that this mode may cause difficulties in knowledge transfer due to the mismatched model capacities. To alleviate the mismatch problem, teacher-student joint training methods, e.g., online distillation, have been proposed, but it always requires expensive computational cost. In this paper, we present a parameter-efficient and student-friendly knowledge distillation method, namely PESF-KD, to achieve efficient and sufficient knowledge transfer by updating relatively few partial parameters. Technically, we first mathematically formulate the mismatch as the sharpness gap between their predictive distributions, where we show such a gap can be narrowed with the appropriate smoothness of the soft label. Then, we introduce an adapter module for the teacher and only update the adapter to obtain soft labels with appropriate smoothness. Experiments on a variety of benchmarks show that PESF-KD can significantly reduce the training cost while obtaining competitive results compared to advanced online distillation methods. Code will be released upon acceptance.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 48 of its 69 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

48 of 69 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
36 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2019When Does Label Smoothing Help?cited by 985
  9. 2019Relational Knowledge Distillationcited by 873
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2019Contrastive Representation Distillationcited by 734
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  13. 2017Deep Mutual Learningcited by 621
  14. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  15. 2019Similarity-Preserving Knowledge Distillationcited by 507
  16. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  17. 2019On the Efficacy of Knowledge Distillationcited by 377
  18. 2019Variational Information Distillation for Knowledge Transfercited by 321
  19. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  20. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  21. 2019Correlation Congruence for Knowledge Distillationcited by 280
  22. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  23. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  24. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  25. 2021Does Knowledge Distillation Really Work?cited by 187
  26. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  27. 2019Online Knowledge Distillation with Diverse Peerscited by 127
  28. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  29. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  30. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  31. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  32. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  33. 2021Distill on the Go: Online knowledge distillation in self-supervised learningcited by 22
  34. 2022Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrievalcited by 19
  35. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  36. 2021Semi-Online Knowledge Distillationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 12; 2025: 15; 2026: 6.

47 papers.

Sort:

By area:

  1. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 79
  2. 2023Visual Tuningcited by 34
  3. 2022Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUEcited by 24
  4. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  5. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 21
  6. 2022Dynamic Contrastive Distillation for Image-Text Retrievalcited by 14
  7. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  8. 2024Federated Distillation: A Surveycited by 14
  9. 2023When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence Systemcited by 13
  10. 2025APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Trainingcited by 12
  11. 2023Student-friendly Knowledge Distillationcited by 11
  12. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
35 more
  1. 2023Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillationcited by 8
  2. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  3. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  4. 2024DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimizationcited by 5
  5. 2024Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detectioncited by 4
  6. 2025Competitive Distillation: A Simple Learning Strategy for Improving Visual Classificationcited by 4
  7. 2025AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMscited by 4
  8. 2023Boosting Residual Networks with Group Knowledgecited by 3
  9. 2024Communication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsificationcited by 3
  10. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  11. 2026FAST: A Synergistic Framework of Attention and State-space Models for Spatiotemporal Traffic Predictioncited by 3
  12. 2026Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learningcited by 3
  13. 2023OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML Systemcited by 2
  14. 2023Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillationcited by 2
  15. 2024iDAT: inverse Distillation Adapter-Tuningcited by 2
  16. 2024Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformercited by 2
  17. 2024Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matchingcited by 2
  18. 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challengescited by 2
  19. 2024Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detectioncited by 1
  20. 2025DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Imagescited by 1
  21. 2025MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Modelscited by 1
  22. 2023One-stop Training of Multiple Capacity Modelscited by 0
  23. 2023Epistemic Graph: A Plug-And-Play Module For Hybrid Representation Learningcited by 0
  24. 2024SPA: Towards A Computational Friendly Cloud-Base and On-Devices Collaboration Seq2seq Personalized Generation with Casual Inferencecited by 0
  25. 2024Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teachercited by 0
  26. 2025Knowledge Distillation with Adapted Weightcited by 0
  27. 2025AgentPose: Progressive Distribution Alignment via Feature Agent for Human Pose Distillationcited by 0
  28. 2025PLD: A Choice-Theoretic List-Wise Knowledge Distillationcited by 0
  29. 2025Explainable Knowledge Distillation for Efficient Medical Image Classificationcited by 0
  30. 2025Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensingcited by 0
  31. 2025SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Modelscited by 0
  32. 2026BicKD: Bilateral Contrastive Knowledge Distillationcited by 0
  33. 2026Generative Diffusion Prior Distillation for Long-Context Knowledge Transfercited by 0
  34. 2026IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compressioncited by 0
  35. 2026KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Rerankingcited by 0