mlsanityData to 14 Oct 2026

Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexity

arXiv 2512.04165 ↗First posted 3 Dec 2025cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Two pressing topics in the theory of deep learning are the interpretation of feature learning (FL) mechanisms and the determination of implicit bias of networks in the rich regime. Current theories of rich FL often appear in the form of high-dimensional non-linear equations, which require computationally intensive numerical solutions. Given the many details that go into defining a deep learning problem, this analytical complexity is a significant and often unavoidable challenge. Here, we propose a powerful heuristic route for predicting the data and width scales at which various patterns of FL emerge. This form of scale analysis is considerably simpler than such exact theories and reproduces the scaling exponents of various known results. In addition, we make novel predictions on complex toy architectures, such as three-layer non-linear networks and attention heads, thus extending the scope of first-principle theories of deep learning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 3 Dec 2025.

Builds on

32 of 52 references.

References · 32

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2022Toy Models of Superpositioncited by 971
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  5. 2019Benign Overfitting in Linear Regressioncited by 549
  6. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  8. 2021Explaining Neural Scaling Lawscited by 363
  9. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  12. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
20 more
  1. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  2. 2020The Case for Bayesian Deep Learningcited by 69
  3. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  4. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  5. 2020Is SGD a Bayesian sampler? Well, almostcited by 50
  6. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  7. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  8. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  9. 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
  10. 2024An exactly solvable model for emergence and scaling laws in the multitask sparse parity problemcited by 22
  11. 2023A Rainbow in Deep Network Black Boxescited by 18
  12. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  13. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  14. 2024High-dimensional learning of narrow neural networkscited by 12
  15. 2019Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspectivecited by 11
  16. 2024Coding schemes in neural networks learning classification taskscited by 11
  17. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  18. 2025Applications of Statistical Field Theory in Deep Learningcited by 9
  19. 2024Demystifying Spectral Bias on Real-World Datacited by 3
  20. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  3. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  4. 2026Width-Robust Learnability in Mean-Field Bayesian Neural Networkscited by 0