mlsanityData to 14 Oct 2026

Sparse Teachers Can Be Dense with Knowledge

arXiv 2210.03923 ↗First posted 8 Oct 2022cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advances in distilling pretrained language models have discovered that, besides the expressiveness of knowledge, the student-friendliness should be taken into consideration to realize a truly knowledgable teacher. Based on a pilot study, we find that over-parameterized teachers can produce expressive yet student-unfriendly knowledge and are thus limited in overall knowledgableness. To remove the parameters that result in student-unfriendliness, we propose a sparse teacher trick under the guidance of an overall knowledgable score for each teacher parameter. The knowledgable score is essentially an interpolation of the expressiveness and student-friendliness scores. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Extensive experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance in comparison with a series of competitive baselines.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 8 Oct 2022.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
27 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2018Neural Network Acceptability Judgmentscited by 958
  6. 2019Are Sixteen Heads Really Better than One?cited by 816
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  8. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  10. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  13. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  14. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  16. 2022Decoupled Knowledge Distillationcited by 388
  17. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  18. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
  19. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  20. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  21. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  22. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  23. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 60
  24. 2016Faster CNNs with Direct Sparse Convolutions and Guided Pruningcited by 39
  25. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  26. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  27. 2022Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Ticketscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 3; 2025: 2.

11 papers.

  1. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  2. 2022XPrompt: Exploring the Extreme of Prompt Tuningcited by 30
  3. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  4. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 25
  5. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  6. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  7. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4
  8. 2023Sparse Contrastive Learning of Sentence Embeddingscited by 1
  9. 2024MoDification: Mixture of Depths Made Easycited by 1
  10. 2023On Elastic Language Modelscited by 0
  11. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0