# A Neural Scaling Law from the Dimension of the Data Manifold

arXiv: https://arxiv.org/abs/2004.10802 · HTML: https://mlsanity.com/paper/2004.10802/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 30 of 52)

- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Language models scale reliably with over-training and on downstream tasks](https://mlsanity.com/paper/2403.08540/) (2024)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Limitations of the NTK for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2206.10012/) (2022)
- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [Scaling Law for Time Series Forecasting](https://mlsanity.com/paper/2405.15124/) (2024)
- [Scaling Laws for Deep Learning](https://mlsanity.com/paper/2108.07686/) (2021)
- [Deriving Neural Scaling Laws from the statistics of natural language](https://mlsanity.com/paper/2602.07488/) (2026)
- [Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks](https://mlsanity.com/paper/2311.12997/) (2023)
- [A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules](https://mlsanity.com/paper/2503.12811/) (2025)
- [Topological Obstructions to Autoencoding](https://mlsanity.com/paper/2102.08380/) (2021)
- [Deconstructing Distributions: A Pointwise Framework of Learning](https://mlsanity.com/paper/2202.09931/) (2022)
- [Hardness of Learning Neural Networks under the Manifold Hypothesis](https://mlsanity.com/paper/2406.01461/) (2024)
- [A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?](https://mlsanity.com/paper/2108.11018/) (2021)
- [Practical Galaxy Morphology Tools from Deep Supervised Representation Learning](https://mlsanity.com/paper/2110.12735/) (2021)
- [Precision Machine Learning](https://mlsanity.com/paper/2210.13447/) (2022)
- [Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition](https://mlsanity.com/paper/2306.14670/) (2023)
- [Intrinsic Entropy of Context Length Scaling in LLMs](https://mlsanity.com/paper/2502.01481/) (2025)
- [Improved Scaling Laws in Linear Regression via Data Reuse](https://mlsanity.com/paper/2506.08415/) (2025)
- [A Resource Model For Neural Scaling Law](https://mlsanity.com/paper/2402.05164/) (2024)
- [An Empirical Study of Scaling Laws for Transfer](https://mlsanity.com/paper/2408.16947/) (2024)
- [Physics of Skill Learning](https://mlsanity.com/paper/2501.12391/) (2025)
- [Exploring the Representation Manifolds of Stable Diffusion Through the Lens of Intrinsic Dimension](https://mlsanity.com/paper/2302.09301/) (2023)

## Abstract (arXiv)

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law L propto N-α in the number of network parameters N. This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension d. This simple theory predicts that the scaling exponents α ≈ 4/d for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of d and α by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.