mlsanityData to 7 Oct 2026

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

arXiv 2409.12903 ↗First posted 19 Sep 2024cs.CL cs.AI cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters can be extremely slow and costly. In contrast, small language models are less expensive to train, but they often cannot achieve the accuracy of large models. In this paper, we explore an intriguing idea to connect these two different regimes: Can we develop a method to initialize large language models using smaller pre-trained models? Will such initialization bring any benefits in terms of training time and final accuracy? In this paper, we introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. Our method ensures that the larger model retains the functionality of the smaller model. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts. We demonstrate that training such an initialized model results in significant savings in terms of GPU hours required for pre-training large language models.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

22 of 26 references.

References · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  11. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  12. 2024The rising costs of training frontier AI modelscited by 91
10 more
  1. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  2. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  3. 2022Staged Training for Transformer Language Modelscited by 49
  4. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  5. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  6. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  7. 2023LEMON: Lossless model expansioncited by 23
  8. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  9. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  10. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 11.

29 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  2. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  3. 2024TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameterscited by 9
  4. 2025Enabling Autoregressive Models to Fill In Masked Tokenscited by 7
  5. 2025Structured Initialization for Vision Transformerscited by 6
  6. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  7. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  8. 2025BeamLoRA: Beam-Constraint Low-Rank Adaptationcited by 3
  9. 2025Motif 2 12.7B technical reportcited by 3
  10. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  11. 2026Autoregressive Models Rival Diffusion Models at ANY-ORDER Generationcited by 3
  12. 2025Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimizationcited by 2
17 more
  1. 2025Grouped Differential Attentioncited by 2
  2. 2026Solar Open 2 Technical Reportcited by 2
  3. 2025Adaptive Neural Quantum States: A Recurrent Neural Network Perspectivecited by 1
  4. 2025GUIDE: Guided Initialization and Distillation of Embeddingscited by 1
  5. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  6. 2026Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Startscited by 1
  7. 2026Mask-Based Priors Are More Persistent than Query-Key Initializationscited by 1
  8. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  9. 2025Advantageous Parameter Expansion Training Makes Better Large Language Modelscited by 0
  10. 2025MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Modelscited by 0
  11. 2025SCALE: Upscaled Continual Learning of Large Language Modelscited by 0
  12. 2026Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Modelscited by 0
  13. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  14. 2026Probing Factual Knowledge Transfer with Training Data Interventionscited by 0
  15. 2026Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillationcited by 0
  16. 2026Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inferencecited by 0
  17. 2026Procedural Core: A Compact Recurrent Initialization for Vision Transformerscited by 0