mlsanityData to 7 Oct 2026

Towards the Law of Capacity Gap in Distilling Language Models

arXiv 2311.07052 ↗First posted 13 Nov 2023cs.CL cs.AI cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Language model (LM) distillation aims at distilling the knowledge in a large teacher LM to a small student one. As a critical issue facing LM distillation, a superior student often arises from a teacher of a relatively small scale instead of a larger one, especially in the presence of substantial capacity gap between the teacher and student. This issue, often referred to as the textitcurse of capacity gap, suggests that there is likely an optimal teacher yielding the best-performing student along the scaling course of the teacher. Consequently, distillation trials on teachers of a wide range of scales are called for to determine the optimal teacher, which becomes computationally intensive in the context of large LMs (LLMs). This paper addresses this critical bottleneck by providing the textitlaw of capacity gap inducted from a preliminary study on distilling a broad range of small-scale (<3B) LMs, where the optimal teacher consistently scales linearly with the student scale across different model and data scales. By extending the law to LLM distillation on a larger scale (7B), we succeed in obtaining versatile LLMs that outperform a wide array of competitors.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 81 of its 121 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

81 of 121 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  13. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  14. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  15. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  16. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  17. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  18. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  19. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  20. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  21. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  22. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  23. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  24. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  25. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  26. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  27. 2019Are Sixteen Heads Really Better than One?cited by 803
  28. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  29. 2016Sequence-Level Knowledge Distillationcited by 761
  30. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  31. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  32. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  33. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  34. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  35. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  36. 2023Textbooks Are All You Needcited by 561
  37. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  38. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 11; 2026: 16.

41 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  4. 2025Distillation Scaling Lawscited by 59
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  6. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  7. 2024INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuningcited by 19
  8. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 13
  9. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 10
  10. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9
  11. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  12. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Modelscited by 7
29 more
  1. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertaintycited by 7
  2. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  3. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  4. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
  5. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  6. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  7. 2024Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysiscited by 5
  8. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  9. 2025Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Startcited by 4
  10. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Modelscited by 4
  11. 2024OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of Instruction Datacited by 3
  12. 2024FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigationcited by 3
  13. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 3
  14. 2026Select to Think: Unlocking SLM Potential with Local Sufficiencycited by 3
  15. 2026Floe: Federated Specialization for Real-Time LLM-SLM Inferencecited by 2
  16. 2024MoDification: Mixture of Depths Made Easycited by 1
  17. 2025Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregationcited by 1
  18. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  19. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Modelscited by 1
  20. 2025LLMs in Mobile Apps: Practices, Challenges, and Opportunitiescited by 0
  21. 2025ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Modelscited by 0
  22. 2026BioAlchemy: Distilling Biological Literature into Reasoning-Ready Reinforcement Learning Training Datacited by 0
  23. 2026Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Modelscited by 0
  24. 2026ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Modelscited by 0
  25. 2026Beyond Imitation: Auditing the Recoverability of Reasoning in Distilled Modelscited by 0
  26. 2026TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignmentcited by 0
  27. 2026Distillation Defenses Easily Break After Reinforcement Learningcited by 0
  28. 2026From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillationcited by 0
  29. 2026The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learningcited by 0