mlsanityData to 14 Oct 2026

Pre-training Distillation for Large Language Models: A Design Space Exploration

arXiv 2410.16215 ↗First posted 21 Oct 2024cs.CL cs.AIcited by 27

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model. Previous work applying KD in the field of large language models (LLMs) typically focused on the post-training phase, where the student LLM learns directly from instructions and corresponding responses generated by the teacher model. In this paper, we extend KD to the pre-training phase of LLMs, named pre-training distillation (PD). We first conduct a preliminary experiment using GLM-4-9B as the teacher LLM to distill a 1.9B parameter student LLM, validating the effectiveness of PD. Considering the key impact factors of distillation, we systematically explore the design space of pre-training distillation across four aspects: logits processing, loss selection, scaling law, and offline or online logits. We conduct extensive experiments to explore the design space of pre-training distillation and find better configurations and interesting conclusions, such as larger student LLMs generally benefiting more from pre-training distillation, while a larger teacher LLM does not necessarily guarantee better results. We hope our exploration of the design space will inform future practices in pre-training distillation.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 45 of its 66 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

45 of 66 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2020Knowledge Distillation: A Surveycited by 1,603
33 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2016Sequence-Level Knowledge Distillationcited by 801
  12. 2019Contrastive Representation Distillationcited by 748
  13. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  14. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  15. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  16. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  17. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  18. 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
  19. 2019Variational Information Distillation for Knowledge Transfercited by 324
  20. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  21. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  22. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  23. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  24. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  25. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  26. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  27. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  28. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 81
  29. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  30. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  31. 2019Adaptive Regularization of Labelscited by 16
  32. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  33. 2023NormKD: Normalized Logits for Knowledge Distillationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 18.

27 papers.

Sort:

By area:

  1. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 175
  2. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 72
  3. 2025Distillation Scaling Lawscited by 67
  4. 2026On-Policy Delta Distillationcited by 15
  5. 2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
  6. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  7. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  8. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  9. 2025Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmissioncited by 5
  10. 2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 5
  11. 2026Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memorycited by 5
  12. 2026Strong Teacher Not Needed? On Distillation in LLM Pretrainingcited by 3
15 more
  1. 2026STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Modelscited by 1
  2. 2026Apertus LLM Family Expansion via Distillation and Quantizationcited by 1
  3. 2025Intelligent Mobile AI-Generated Content Services via Interactive Prompt Engineering and Dynamic Service Provisioningcited by 0
  4. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillationcited by 0
  5. 2025Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Modelscited by 0
  6. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translationcited by 0
  7. 2026A Study on Hidden Layer Distillation for Large Language Model Pre-Trainingcited by 0
  8. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0
  9. 2026Let the Data Decide: Supervision Analysis, Capability Trade-offs, and Adaptive Objective Routing in Continued Pre-Training via Off-Policy Distillationcited by 0
  10. 2026Matryoshka Language Model Suitescited by 0
  11. 2026ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Modelscited by 0
  12. 2026ACLArena: Agent Continue Learning in Multi-stage Post-trainingcited by 0
  13. 2026ReTaCo: Residual-Target Control for On-Policy Distillationcited by 0
  14. 2026The Gradient That Top-k Distillation Never Computescited by 0
  15. 2026DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Losscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  2. 2026Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
  3. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Models
  4. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
  5. 2023Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection
  6. 2025Less is More: Selective Reflection for Compatible and Efficient Knowledge Distillation in Large Language Models
  7. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  8. 2021Dynamic Knowledge Distillation for Pre-trained Language Models