# Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

arXiv: https://arxiv.org/abs/2409.12903 · HTML: https://mlsanity.com/paper/2409.12903/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [The rising costs of training frontier AI models](https://mlsanity.com/paper/2405.21015/) (2024)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [bert2BERT: Towards Reusable Pretrained Language Models](https://mlsanity.com/paper/2110.07143/) (2021)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training](https://mlsanity.com/paper/2405.15319/) (2024)
- [Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup](https://mlsanity.com/paper/2011.13635/) (2020)
- [Weight subcloning: direct initialization of transformers using larger pretrained ones](https://mlsanity.com/paper/2312.09299/) (2023)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [FLM-101B: An Open LLM and How to Train It with 100K Budget](https://mlsanity.com/paper/2309.03852/) (2023)
- [Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective](https://mlsanity.com/paper/2310.11451/) (2023)
- [Landscape-Aware Growing: The Power of a Little LAG](https://mlsanity.com/paper/2406.02469/) (2024)

## Built on by (most cited first, 29 of 29)

- [Scaling Diffusion Language Models via Adaptation from Autoregressive Models](https://mlsanity.com/paper/2410.17891/) (2024)
- [A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs](https://mlsanity.com/paper/2410.18779/) (2024)
- [TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters](https://mlsanity.com/paper/2410.23168/) (2024)
- [Enabling Autoregressive Models to Fill In Masked Tokens](https://mlsanity.com/paper/2502.06901/) (2025)
- [A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthiness](https://mlsanity.com/paper/2510.13890/) (2025)
- [Structured Initialization for Vision Transformers](https://mlsanity.com/paper/2505.19985/) (2025)
- [LESA: Learnable LLM Layer Scaling-Up](https://mlsanity.com/paper/2502.13794/) (2025)
- [Motif 2 12.7B technical report](https://mlsanity.com/paper/2511.07464/) (2025)
- [Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation](https://mlsanity.com/paper/2601.13228/) (2026)
- [BeamLoRA: Beam-Constraint Low-Rank Adaptation](https://mlsanity.com/paper/2502.13604/) (2025)
- [Grouped Differential Attention](https://mlsanity.com/paper/2510.06949/) (2025)
- [Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation](https://mlsanity.com/paper/2512.10772/) (2025)
- [Solar Open 2 Technical Report](https://mlsanity.com/paper/2607.20062/) (2026)
- [Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization](https://mlsanity.com/paper/2504.10735/) (2025)
- [Adaptive Neural Quantum States: A Recurrent Neural Network Perspective](https://mlsanity.com/paper/2507.18700/) (2025)
- [GUIDE: Guided Initialization and Distillation of Embeddings](https://mlsanity.com/paper/2510.06502/) (2025)
- [Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better](https://mlsanity.com/paper/2602.05393/) (2026)
- [Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Starts](https://mlsanity.com/paper/2604.04281/) (2026)
- [Making Transformers Logical](https://mlsanity.com/paper/2608.00418/) (2026)
- [ResidualDroppath: Enhancing Feature Reuse over Residual Connections](https://mlsanity.com/paper/2411.09475/) (2024)
- [Advantageous Parameter Expansion Training Makes Better Large Language Models](https://mlsanity.com/paper/2505.24241/) (2025)
- [MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Models](https://mlsanity.com/paper/2510.17519/) (2025)
- [SCALE: Upscaled Continual Learning of Large Language Models](https://mlsanity.com/paper/2511.03270/) (2025)
- [Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models](https://mlsanity.com/paper/2605.07783/) (2026)
- [When is Warmstarting Effective for Scaling Language Models?](https://mlsanity.com/paper/2605.13405/) (2026)
- [Probing Factual Knowledge Transfer with Training Data Interventions](https://mlsanity.com/paper/2609.01341/) (2026)
- [Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation](https://mlsanity.com/paper/2609.02006/) (2026)
- [Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference](https://mlsanity.com/paper/2609.05275/) (2026)
- [Procedural Core: A Compact Recurrent Initialization for Vision Transformers](https://mlsanity.com/paper/2609.37631/) (2026)

## Abstract (arXiv)

The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters can be extremely slow and costly. In contrast, small language models are less expensive to train, but they often cannot achieve the accuracy of large models. In this paper, we explore an intriguing idea to connect these two different regimes: Can we develop a method to initialize large language models using smaller pre-trained models? Will such initialization bring any benefits in terms of training time and final accuracy? In this paper, we introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. Our method ensures that the larger model retains the functionality of the smaller model. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts. We demonstrate that training such an initialized model results in significant savings in terms of GPU hours required for pre-training large language models.