On the Smallness of the Large Language Models Scaling Exponents
Not on the map. Why?
Abstract (arXiv)
We discuss reasons why the scaling exponents of current Large Language Models (LLMs) applications are indicating an unsustainable regime in terms of energy resources. We further show that attributing the smallness of such exponents to a numerical bias due to the neglect of a non-zero value of the loss function in the limit of infinite data (``pedestal effect") does not remove the unsustainability issue. Finally, the effects of the smoothness (roughness) of the data on the scaling exponents is commented upon based on an analogy with phenomenological models of fluid turbulence.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 24 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
10 of 24 references.
References · 10
- 2017Attention Is All You Needcited by 60,361
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2021Explaining Neural Scaling Lawscited by 363
- 2019Intrinsic dimension of data representations in deep neural networkscited by 282
- 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 47
- 2025Superposition Yields Robust Neural Scalingcited by 31
- 2025The wall confronting large language modelscited by 8
- 2025A note on the physical interpretation of neural PDE'scited by 7
- 2025A new kind of sciencecited by 1
Built on by
Not cited yet.