mlsanityData to 7 Oct 2026

Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

arXiv 2606.25008 ↗First posted 23 Jun 2026cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power laws are fixed by generic mechanisms: a one-third time scaling due to the strong nonlinearity of Softmax, an inverse width scaling due to representational superposition, and an inverse depth scaling due to ensemble averaging of Transformer layers. These mechanisms are robust to a wide range of data structures and architectural details, placing current large language models in a universality class with fixed exponents. The coefficients, however, are expected to be sensitive to data and architecture details, and directly determine practical quantities such as the optimal model shape and the compute-optimal frontier. We therefore argue that understanding the coefficients is the key to near-term performance improvements, and that a closer examination of the current universality class may reveal pathways to better universality classes.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 45 of its 50 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

45 of 50 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2022Toy Models of Superpositioncited by 928
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2025Muon is Scalable for LLM Trainingcited by 381
33 more
  1. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  2. 2021Explaining Neural Scaling Lawscited by 353
  3. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  4. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  5. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  6. 2023The Quantization Model of Neural Scalingcited by 137
  7. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  8. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  9. 2022A Solvable Model of Neural Scaling Lawscited by 101
  10. 2024Looped Transformers for Length Generalizationcited by 91
  11. 2021Learning Curve Theorycited by 79
  12. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  13. 2024Chinchilla Scaling: A replication attemptcited by 65
  14. 2025The Curse of Depth in Large Language Modelscited by 58
  15. 2026Attention Residualscited by 57
  16. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  17. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  18. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  19. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  20. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  21. 2025Superposition Yields Robust Neural Scalingcited by 30
  22. 2024An exactly solvable model for emergence and scaling laws in the multitask sparse parity problemcited by 21
  23. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  24. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  25. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  26. 2024When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Modelscited by 12
  27. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  28. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  29. 2025Physics of Skill Learningcited by 8
  30. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
  31. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  32. 2025Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Modelscited by 5
  33. 2026Superposition unifies power-law training dynamicscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Lawscited by 0
  2. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0
  3. 2026Emergent phases of superposition: from partial to full representationcited by 0
  4. 2026Emergent Inverse-Depth Scaling From Nonlinearity In Attentioncited by 0