mlsanityData to 7 Oct 2026

MiniDisc: Minimal Distillation Schedule for Language Model Compression

arXiv 2205.14570 ↗First posted 29 May 2022cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent studies have uncovered that language model distillation is less effective when facing a large capacity gap between the teacher and the student, and introduced teacher assistant-based distillation to bridge the gap. As a connection, the scale and the performance of the teacher assistant is of vital importance to bring the knowledge from the teacher to the student. However, existing teacher assistant-based methods require maximally many trials before scheduling an optimal teacher assistant. To this end, we propose a minimal distillation schedule (MiniDisc) for scheduling the optimal teacher assistant in minimally one trial. In particular, motivated by the finding that the performance of the student is positively correlated to the scale-performance tradeoff of the teacher assistant, MiniDisc is designed with a λ-tradeoff to measure the optimality of the teacher assistant without trial distillation to the student. MiniDisc then can schedule the optimal teacher assistant with the best λ-tradeoff in a sandwich framework. MiniDisc is evaluated with an extensive set of experiments on GLUE. Experimental results demonstrate the improved efficiency our MiniDisc compared to several state-of-the-art baselines. We further apply MiniDisc to a language model with billions of parameters and show its scalability.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 29 May 2022.

Builds on

41 of 57 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
29 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  3. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  10. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  13. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  14. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  15. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  16. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  17. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  18. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  19. 2019Pareto Multi-Task Learningcited by 224
  20. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  21. 2021Block Pruning For Faster Transformerscited by 176
  22. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  23. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  24. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  25. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  26. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  27. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  28. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  29. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 0; 2026: 1.

5 papers.

  1. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  2. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  3. 2024MoDification: Mixture of Depths Made Easycited by 1
  4. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Modelscited by 1
  5. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0