mlsanityData to 14 Oct 2026

Don't be lazy: CompleteP enables compute-efficient deep transformers

arXiv 2505.01618 ↗First posted 2 May 2025cs.LG cs.AIcited by 78

Not on the map. Why?

Abstract (arXiv)

We study compute efficiency of LLM training when using different parameterizations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), or accept sub-optimal training when re-tuning is prohibitive. Even when they achieve HP transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, we identify and adopt the parameterization we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art. All experiments were run on Cerebras CS-3 systems. A minimal implementation is available at https://github.com/EleutherAI/nanoGPT-mup/tree/completep.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Builds on 85 of its 110 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

85 of 110 references.

References · 85

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2025Gemma 3 Technical Reportcited by 1,824
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  13. 2018On Lazy Training in Differentiable Programmingcited by 675
  14. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  15. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  16. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  17. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  18. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  19. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  20. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  21. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  22. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  23. 20242 OLMo 2 Furiouscited by 340
  24. 2018An Empirical Model of Large-Batch Trainingcited by 281
  25. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  26. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  27. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  28. 2016Deep Information Propagationcited by 195
  29. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  30. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  31. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  32. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  33. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  34. 2021The Principles of Deep Learning Theorycited by 173
  35. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  36. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  37. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  38. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 56.

78 papers.

Sort:

By area:

  1. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 43
  2. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 35
  3. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  4. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 27
  5. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  6. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  7. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  8. 2025Exact Expressive Power of Transformers with Paddingcited by 19
  9. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 17
  10. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 16
  11. 2026A Defense of the Quadratic Modelcited by 14
  12. 2026Hyperparameter Transfer with Mixture-of-Expert Layerscited by 13
38 more
  1. 2026Sparse Layers are Critical to Scaling Looped Language Modelscited by 11
  2. 2026On the Residual Scaling of Looped Transformers: Stability and Transferabilitycited by 10
  3. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  4. 2025Optimal Scaling Needs Optimal Normcited by 8
  5. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  6. 2026Scaling Categorical Flow Mapscited by 8
  7. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 7
  8. 2026Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcited by 7
  9. 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 7
  10. 2025On the Origin of Algorithmic Progress in AIcited by 6
  11. 2025Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysiscited by 6
  12. 2026Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Modelcited by 6
  13. 2026The Design Space of Tri-Modal Masked Diffusion Modelscited by 6
  14. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  15. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
  16. 2026How to Set the Learning Rate for Large-Scale Pre-training?cited by 5
  17. 2026Homogenized Transformerscited by 5
  18. 2026When Does Sparsity Mitigate the Curse of Depth in LLMscited by 4
  19. 2025μ-Parametrization for Mixture of Expertscited by 3
  20. 2025Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regimecited by 3
  21. 2025A Proof of Learning Rate Transfer under μPcited by 3
  22. 2025Controlling changes to attention logitscited by 3
  23. 2026Learning Rate Transfer in Normalized Transformerscited by 3
  24. 2026One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMscited by 3
  25. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  26. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3
  27. 2025Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMscited by 2
  28. 2025MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updatescited by 2
  29. 2026Learnable Multipliers: Freeing the Scale of Language Model Matrix Layerscited by 2
  30. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  31. 2026Configuration-to-Performance Scaling Law with Neural Ansatzcited by 2
  32. 2026Tight Complexity of Muon under Heavy-Tailed Noisecited by 2
  33. 2026Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Ratecited by 2
  34. 2026MuCon: Clipped Muon Updates for LLM Trainingcited by 2
  35. 2026Statistical Properties of Training & Generalizationcited by 2
  36. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  37. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  38. 2026Small-Scale Experiments: Are We There Yet?cited by 2

50 most cited of 78 All 78 (JSON)