mlsanityData to 14 Oct 2026

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection

arXiv 2302.00444 ↗First posted 1 Feb 2023cs.CLcited by 12

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation addresses the problem of transferring knowledge from a teacher model to a student model. In this process, we typically have multiple types of knowledge extracted from the teacher model. The problem is to make full use of them to train the student model. Our preliminary study shows that: (1) not all of the knowledge is necessary for learning a good student model, and (2) knowledge distillation can benefit from certain knowledge at different training steps. In response to these, we propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. In addition, we offer a refinement of the training algorithm to ease the computational burden. Experimental results on the GLUE datasets show that our method outperforms several strong knowledge distillation baselines significantly.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 25 of its 44 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

25 of 44 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
13 more
  1. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  2. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  3. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  4. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  5. 2021Does Knowledge Distillation Really Work?cited by 190
  6. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  7. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  8. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 63
  9. 2021Annealing Knowledge Distillationcited by 61
  10. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  11. 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
  12. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  13. 2019An Investigation of Transfer Learning-Based Sentiment Analysis in Japanesecited by 7

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4; 2025: 5.

12 papers.

  1. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  2. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  3. 2024Collective Critics for Creative Story Generationcited by 22
  4. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  5. 2023Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillationcited by 8
  6. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  7. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  8. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  9. 2024LRHP: Learning Representations for Human Preferences via Preference Pairscited by 5
  10. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  11. 2025Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)cited by 2
  12. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration
  2. 2025Knowledge Distillation with Training Wheels
  3. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  4. 2023Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication
  5. 2025Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models
  6. 2026Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
  7. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation
  8. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation