mlsanityData to 7 Oct 2026

BERT Learns to Teach: Knowledge Distillation with Meta Learning

arXiv 2106.04570 ↗First posted 8 Jun 2021cs.LG cs.AI cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is fixed during training. We show the teacher network can learn to better transfer knowledge to the student network (i.e., learning to teach) with the feedback from the performance of the distilled student network in a meta learning framework. Moreover, we introduce a pilot update mechanism to improve the alignment between the inner-learner and meta-learner in meta learning algorithms that focus on an improved inner-learner. Experiments on various benchmarks show that MetaDistil can yield significant improvements compared with traditional KD algorithms and is less sensitive to the choice of different student capacity and hyperparameters, facilitating the use of KD on different tasks and models.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

43 of 63 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
31 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2019Relational Knowledge Distillationcited by 873
  6. 2019Contrastive Representation Distillationcited by 734
  7. 2017Deep Mutual Learningcited by 621
  8. 2016Learning to learn by gradient descent by gradient descentcited by 616
  9. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  10. 2019Similarity-Preserving Knowledge Distillationcited by 507
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2020Meta Pseudo Labelscited by 372
  13. 2019Variational Information Distillation for Knowledge Transfercited by 321
  14. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  15. 2019Correlation Congruence for Knowledge Distillationcited by 280
  16. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  17. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 255
  18. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  19. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  20. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  21. 2020Characterising Bias in Compressed Modelscited by 162
  22. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  23. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  24. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  25. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  26. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  27. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  28. 2020Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domainscited by 33
  29. 2020MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillationcited by 18
  30. 2021Meta Back-translationcited by 16
  31. 2021Improving Sequence-to-Sequence Pre-training via Sequence Span Rewritingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 17; 2024: 14; 2025: 11; 2026: 4.

61 papers.

Sort:

By area:

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  3. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  4. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  5. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  6. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  7. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  8. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  9. 2025Efficient Agents: Building Effective Agents While Reducing Costcited by 32
  10. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  11. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  12. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
38 more
  1. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  2. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  3. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  4. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  5. 2022PROD: Progressive Distillation for Dense Retrievalcited by 20
  6. 2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
  7. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  8. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  9. 2023Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learningcited by 16
  10. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  11. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  12. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  13. 2022Sparse Teachers Can Be Dense with Knowledgecited by 11
  14. 2023Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Modelscited by 11
  15. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  16. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9
  17. 2023Weight-Inherited Distillation for Task-Agnostic BERT Compressioncited by 8
  18. 2023Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillationcited by 8
  19. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  20. 2022Meta Self-Refinement for Robust Learning with Weak Supervisioncited by 7
  21. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Modelscited by 6
  22. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  23. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  24. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  25. 2026DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillationcited by 5
  26. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inferencecited by 4
  27. 2023Meta-Tsallis-Entropy Minimization: A New Self-Training Approach for Domain Adaptation on Text Classificationcited by 4
  28. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  29. 2023Persona-aware Generative Model for Code-mixed Languagecited by 3
  30. 2024Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Modelscited by 3
  31. 2024Training the Untrainable: Introducing Inductive Bias via Representational Alignmentcited by 3
  32. 2025Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflectioncited by 2
  33. 2022Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolutioncited by 1
  34. 2023Dialogue State Distillation Network with Inter-slot Contrastive Learning for Dialogue State Trackingcited by 1
  35. 2023Multilingual Sentence-Level Semantic Search using Meta-Distillation Learningcited by 1
  36. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Expertscited by 1
  37. 2025What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry Viewcited by 1
  38. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1

50 most cited of 61 All 61 (JSON)