mlsanityData to 7 Oct 2026

A Neural Scaling Law from the Dimension of the Data Manifold

arXiv 2004.10802 ↗First posted 22 Apr 2020cs.LG stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law L propto N-α in the number of network parameters N. This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension d. This simple theory predicts that the scaling exponents α ≈ 4/d for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of d and α by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 6; 2023: 6; 2024: 12; 2025: 8; 2026: 8.

48 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  4. 2021Explaining Neural Scaling Lawscited by 353
  5. 2021Scaling Laws for Transfercited by 265
  6. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  7. 2021Learning Curve Theorycited by 79
  8. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  9. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  10. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  11. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  12. 2024Scaling Law for Time Series Forecastingcited by 31
36 more
  1. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  2. 2021Scaling Laws for Deep Learningcited by 25
  3. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  4. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  5. 2021Topological Obstructions to Autoencodingcited by 20
  6. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  7. 2024Hardness of Learning Neural Networks under the Manifold Hypothesiscited by 18
  8. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  9. 2021A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?cited by 14
  10. 2021Practical Galaxy Morphology Tools from Deep Supervised Representation Learningcited by 14
  11. 2022Precision Machine Learningcited by 14
  12. 2023Improved Bayes Risk Can Yield Reduced Social Welfare Under Competitioncited by 14
  13. 2025Intrinsic Entropy of Context Length Scaling in LLMscited by 14
  14. 2024An Empirical Study of Scaling Laws for Transfercited by 9
  15. 2024A Resource Model For Neural Scaling Lawcited by 8
  16. 2025Physics of Skill Learningcited by 8
  17. 2023Exploring the Representation Manifolds of Stable Diffusion Through the Lens of Intrinsic Dimensioncited by 7
  18. 2024Survival of the Fittest Representation: A Case Study with Modular Additioncited by 7
  19. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  20. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 7
  21. 2023A Neural Scaling Law from Lottery Ticket Ensemblingcited by 5
  22. 2023Scaling Laws in Jet Classificationcited by 5
  23. 2025Dynamic Reflections: Probing Video Representations with Text Alignmentcited by 5
  24. 2026Scaling laws for amplitude surrogatescited by 5
  25. 2021Tensor network to learn the wavefunction of datacited by 4
  26. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelinescited by 4
  27. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  28. 2022Nonlinear Initialization Methods for Low-Rank Neural Networkscited by 3
  29. 2025Optimal Look-back Horizon for Time Series Forecasting in Federated Learningcited by 2
  30. 2024KAT to KANs: A Review of Kolmogorov-Arnold Networks and the Neural Leap Forwardcited by 1
  31. 2026Superposition unifies power-law training dynamicscited by 1
  32. 2026Effective Frontiers: A Unification of Neural Scaling Lawscited by 1
  33. 2026Towards Engineering Scaling Laws with Pretraining Data Compositioncited by 1
  34. 2026Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Modelscited by 0
  35. 2026Structure and Scale in Simplicial Sequence Modellingcited by 0
  36. 2026Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scalingcited by 0