mlsanityData to 7 Oct 2026

Deriving Neural Scaling Laws from the statistics of natural language

arXiv 2602.07488 ↗First posted 7 Feb 2026cs.LG cs.AI stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 47 of its 68 references that are in the map. First posted on arXiv 7 Feb 2026.

Builds on

47 of 68 references.

References · 47

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
35 more
  1. 2021Explaining Neural Scaling Lawscited by 353
  2. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  5. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  6. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  7. 2023The Quantization Model of Neural Scalingcited by 137
  8. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  9. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  10. 2022A Solvable Model of Neural Scaling Lawscited by 101
  11. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  12. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  13. 2021Learning Curve Theorycited by 79
  14. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  15. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  16. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  17. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48
  18. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  19. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
  20. 2016Criticality in Formal Languages and Statistical Physicscited by 33
  21. 2023Scaling Laws for Associative Memoriescited by 30
  22. 2025Superposition Yields Robust Neural Scalingcited by 30
  23. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 29
  24. 2024A polar coordinate system represents syntax in large language modelscited by 24
  25. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
  26. 2025Pre-training under infinite computecited by 24
  27. 2024An exactly solvable model for emergence and scaling laws in the multitask sparse parity problemcited by 21
  28. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 19
  29. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  30. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  31. 2026Differential syntactic and semantic encoding in LLMscited by 6
  32. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  33. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  34. 2025From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesiscited by 2
  35. 2025Large language models and the entropy of Englishcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  2. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  3. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 4
  4. 2026Unlearnable phases of mattercited by 2
  5. 2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 2
  6. 2026Statistical Properties of Training & Generalizationcited by 2
  7. 2026Semantic Chunking and the Entropy of Natural Languagecited by 1
  8. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 1
  9. 2026Smooth Scaling Laws Hide Stepwise Token Learningcited by 1
  10. 2026Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Datacited by 0
  11. 2026Asymmetric Scaling Laws from Sparse Featurescited by 0
  12. 2026Sampling Data with Chains of Forward-Backward Diffusion Stepscited by 0
6 more
  1. 2026Neuron Populations Exhibit Divergent Selectivity with Scalecited by 0
  2. 2026Critical Percolation as a Synthetic Data Model for Interpretabilitycited by 0
  3. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0
  4. 2026Limits of Reliability and Scaling in Language Modelscited by 0
  5. 2026Do Quantum Models Scale Like LLMs?cited by 0
  6. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0