mlsanityData to 14 Oct 2026

Lifting the Curse of Capacity Gap in Distilling Language Models

arXiv 2305.12129 ↗First posted 20 May 2023cs.CL cs.LGcited by 25

On the map: Mixture-of-experts → MoE large language models

also in 1 more
Abstract (arXiv)

Pretrained language models (LMs) have shown compelling performance on various downstream tasks, but unfortunately they require a tremendous amount of inference compute. Knowledge distillation finds a path to compress LMs to small ones with a teacher-student paradigm. However, when the capacity gap between the teacher and the student is large, a curse of capacity gap appears, invoking a deficiency in distilling LMs. While a few studies have been carried out to fill the gap, the curse is not yet well tackled. In this paper, we aim at lifting the curse of capacity gap via enlarging the capacity of the student without notably increasing the inference compute. Largely motivated by sparse activation regime of mixture of experts (MoE), we propose a mixture of minimal experts (MiniMoE), which imposes extra parameters to the student but introduces almost no additional inference compute. Experimental results on GLUE and CoNLL demonstrate the curse of capacity gap is lifted by the magic of MiniMoE to a large extent. MiniMoE also achieves the state-of-the-art performance at small FLOPs compared with a range of competitive baselines. With a compression rate as much as ~50×, MiniMoE preserves ~95% GLUE score of the teacher.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 55 of its 76 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

55 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation15 Apr
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  10. 2021Dynamic Neural Networks: A Surveycited by 408
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  12. 2022Decoupled Knowledge Distillationcited by 388
  13. 2019On the Efficacy of Knowledge Distillationcited by 381
  14. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  15. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  16. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  17. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  18. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  19. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  20. 2021Hash Layers For Large Sparse Modelscited by 237
  21. 2015Unifying distillation and privileged informationcited by 233
  22. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  23. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  24. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  25. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  26. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  27. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  28. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  29. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  30. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  31. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 60
  32. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  33. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  34. 2015Resiliency of Deep Neural Networks under Quantizationcited by 35
  35. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  36. 2020BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distancecited by 27
  37. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  38. 2022Sparse Teachers Can Be Dense with Knowledgecited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 6; 2025: 10; 2026: 2.

25 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  3. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  4. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  5. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 42
  6. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  7. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 32
  8. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  9. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
  10. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  11. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  12. 2024Don't Throw Away Data: Better Sequence Knowledge Distillationcited by 9
13 more
  1. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9
  2. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  3. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  4. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  5. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 5
  6. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4
  7. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 3
  8. 2025Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilitiescited by 2
  9. 2025AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distributioncited by 2
  10. 2024MoDification: Mixture of Depths Made Easycited by 1
  11. 2023On Elastic Language Modelscited by 0
  12. 2025LLMs in Mobile Apps: Practices, Challenges, and Opportunitiescited by 0
  13. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  2. 2026Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings
  3. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  4. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
  6. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Models
  7. 2026Strong Teacher Not Needed? On Distillation in LLM Pretraining
  8. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models