mlsanityData to 7 Oct 2026

Patient Knowledge Distillation for BERT Model Compression

arXiv 1908.09355 ↗First posted 25 Aug 2019cs.CL cs.LGcited by 502

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: (i) PKD-Last: learning from the last k layers; and (ii) PKD-Skip: learning from every k layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy.

Key facts. Cited by 502 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 114; 2022: 88; 2023: 100; 2024: 82; 2025: 67; 2026: 51.

502 papers.

Sort:

By area:

  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  4. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2022Confident Adaptive Language Modelingcited by 297
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  8. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  9. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  10. 2022Language model compression with weighted low-rank factorizationcited by 192
  11. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  12. 2021Block Pruning For Faster Transformerscited by 176
38 more
  1. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  2. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  3. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  4. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  5. 2021Learned Token Pruning for Transformerscited by 138
  6. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  7. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  8. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  9. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  10. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  11. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  12. 2024Knowledge Fusion of Large Language Modelscited by 110
  13. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  14. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  15. 2021Trustworthy AI: A Computational Perspectivecited by 93
  16. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  17. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  18. 2021Vision Transformer Pruningcited by 77
  19. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  20. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  21. 2021Annealing Knowledge Distillationcited by 61
  22. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  23. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillationcited by 60
  24. 2022Factorizing Knowledge in Neural Networkscited by 59
  25. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  26. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  27. 2021Go Wider Instead of Deepercited by 56
  28. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  29. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  30. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  31. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  32. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  33. 2021A Survey on Green Deep Learningcited by 52
  34. 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
  35. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  36. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  37. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  38. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45

50 most cited of 502 All 502 (JSON)