mlsanityData to 7 Oct 2026

GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model

arXiv 2306.06629 ↗First posted 11 Jun 2023cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deployment of knowledge distillation systems faces great challenges in real-world industrial-strength applications, which require the use of complex distillation methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the switching of methods. To overcome these challenges, we propose GKD, a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. With GKD, developers can build larger distillation models on memory-limited GPUs and easily switch and combine different distillation methods within a single framework. Experimental results show that GKD can support the distillation of at least 100B-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

33 of 42 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
21 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  3. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  4. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  5. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  6. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  7. 2019Knowledge Distillation from Internal Representationscited by 87
  8. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  9. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  10. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 62
  11. 2021Annealing Knowledge Distillationcited by 61
  12. 2019Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering Systemcited by 43
  13. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 39
  14. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  15. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  16. 2020LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understandingcited by 27
  17. 2020TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processingcited by 25
  18. 2021Causal Distillation for Language Modelscited by 19
  19. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  20. 2021Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillationcited by 6
  21. 2022Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 8.

15 papers.

Sort:

By area:

  1. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  2. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxescited by 28
  3. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  4. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debatecited by 12
  5. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignmentcited by 10
  6. 2024Reliable Model Watermarking: Defending Against Theft without Compromising on Evasioncited by 9
  7. 2026VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillationcited by 9
  8. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  9. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 3
  10. 2026X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignmentcited by 3
  11. 2024QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Modelscited by 2
  12. 2026DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Modelscited by 2
3 more
  1. 2026LATS: Large Language Model Assisted Teacher-Student Framework for Multi-Agent Reinforcement Learning in Traffic Signal Controlcited by 1
  2. 2026Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systemscited by 0
  3. 2026Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Updatecited by 0