# On the Smallness of the Large Language Models Scaling Exponents

arXiv: https://arxiv.org/abs/2606.24504 · HTML: https://mlsanity.com/paper/2606.24504/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models](https://mlsanity.com/paper/2507.06952/) (2025)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [The wall confronting large language models](https://mlsanity.com/paper/2507.19703/) (2025)
- [A note on the physical interpretation of neural PDE's](https://mlsanity.com/paper/2502.06739/) (2025)
- [A new kind of science](https://mlsanity.com/paper/2601.00849/) (2025)

## Abstract (arXiv)

We discuss reasons why the scaling exponents of current Large Language Models (LLMs) applications are indicating an unsustainable regime in terms of energy resources. We further show that attributing the smallness of such exponents to a numerical bias due to the neglect of a non-zero value of the loss function in the limit of infinite data (``pedestal effect") does not remove the unsustainability issue. Finally, the effects of the smoothness (roughness) of the data on the scaling exponents is commented upon based on an analogy with phenomenological models of fluid turbulence.