mlsanityData to 14 Oct 2026

Gradient Knowledge Distillation for Pre-trained Language Models

arXiv 2211.01071 ↗First posted 2 Nov 2022cs.CL cs.LGcited by 5

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) is an effective framework to transfer knowledge from a large-scale teacher to a compact yet well-performing student. Previous KD practices for pre-trained language models mainly transfer knowledge by aligning instance-wise outputs between the teacher and student, while neglecting an important knowledge source, i.e., the gradient of the teacher. The gradient characterizes how the teacher responds to changes in inputs, which we assume is beneficial for the student to better approximate the underlying mapping function of the teacher. Therefore, we propose Gradient Knowledge Distillation (GKD) to incorporate the gradient alignment objective into the distillation process. Experimental results show that GKD outperforms previous KD methods regarding student performance. Further analysis shows that incorporating gradient knowledge makes the student behave more consistently with the teacher, improving the interpretability greatly.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 23 of its 31 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application9 Feb

Other references · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  12. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
10 more
  1. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  2. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  3. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  4. 2018Knowledge Transfer with Jacobian Matchingcited by 76
  5. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  6. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  7. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  8. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  9. 2019Saliency-driven Word Alignment Interpretation for Neural Machine Translationcited by 27
  10. 2019Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillationcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023On the Opportunities of Green Computing: A Surveycited by 7
  2. 2023Indirect Gradient Matching for Adversarial Robust Distillationcited by 6
  3. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  4. 2024Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insightscited by 4
  5. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
  2. 2023Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication
  3. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration
  4. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  5. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection
  6. 2026Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
  7. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines
  8. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models