mlsanityData to 14 Oct 2026

How Feature Learning Can Improve Neural Scaling Laws

arXiv 2409.17858 ↗First posted 26 Sep 2024stat.ML cond-mat.dis-nn cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

We develop a solvable model of neural scaling laws beyond the kernel limit. Theoretical analysis of this model shows how performance scales with model size, training time, and the total amount of available data. We identify three scaling regimes corresponding to varying task difficulties: hard, easy, and super easy tasks. For easy and super-easy target functions, which lie in the reproducing kernel Hilbert space (RKHS) defined by the initial infinite-width Neural Tangent Kernel (NTK), the scaling exponents remain unchanged between feature learning and kernel regime models. For hard tasks, defined as those outside the RKHS of the initial NTK, we demonstrate both analytically and empirically that feature learning can improve scaling with training time and compute, nearly doubling the exponent for hard tasks. This leads to a different compute optimal strategy to scale parameters and training time in the feature learning regime. We support our finding that feature learning improves the scaling law for hard tasks but not for easy and super-easy tasks with experiments of nonlinear MLPs fitting functions with power-law Fourier spectra on the circle and CNNs learning vision tasks.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 55 of its 75 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

55 of 75 references.

References · 55

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  7. 2018On Lazy Training in Differentiable Programmingcited by 675
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 2023Scaling Data-Constrained Language Modelscited by 403
  12. 2021Explaining Neural Scaling Lawscited by 363
38 more
  1. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  2. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  3. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  6. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  7. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  9. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  10. 2022A Solvable Model of Neural Scaling Lawscited by 116
  11. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  12. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  13. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  14. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  15. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  16. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  17. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  18. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  19. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  20. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  21. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  22. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  23. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  24. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  25. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  26. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 42
  27. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  28. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 41
  29. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  30. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
  31. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 35
  32. 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
  33. 2020Geometric compression of invariant manifolds in neural netscited by 28
  34. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 26
  35. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  36. 2020Towards NNGP-guided Neural Architecture Searchcited by 24
  37. 2022Infinite-width limit of deep linear neural networkscited by 20
  38. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 22; 2026: 27.

56 papers.

Sort:

By area:

  1. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  2. 2025Superposition Yields Robust Neural Scalingcited by 31
  3. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  4. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 22
  5. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  6. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  7. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  8. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 19
  9. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 16
  10. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 16
  11. 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
  12. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
38 more
  1. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  2. 2025Improved Scaling Laws in Linear Regression via Data Reusecited by 11
  3. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 10
  4. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 10
  5. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  6. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  7. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 9
  8. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 9
  9. 2026Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcited by 7
  10. 2025Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Modelscited by 6
  11. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 6
  12. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  13. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 5
  14. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  15. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 3
  16. 2026A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classificationcited by 3
  17. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2
  18. 2025Evaluating the Robustness of Chinchilla Compute-Optimal Scalingcited by 2
  19. 2026The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learningcited by 2
  20. 2026Statistical Properties of Training & Generalizationcited by 2
  21. 2026How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponentscited by 2
  22. 2024No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditionscited by 1
  23. 2025Corner Gradient Descentcited by 1
  24. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1
  25. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  26. 2026Superposition unifies power-law training dynamicscited by 1
  27. 2026Effective Frontiers: A Unification of Neural Scaling Lawscited by 1
  28. 2026Asymmetric Scaling Laws from Sparse Featurescited by 1
  29. 2026Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tailcited by 1
  30. 2026Explaining Data Mixing Scaling Lawscited by 1
  31. 2025Uncertainty Quantification From Scaling Laws in Deep Neural Networkscited by 0
  32. 2025Data Curation Through the Lens of Spectral Dynamics: Static Limits, Dynamic Acceleration, and Practical Oraclescited by 0
  33. 2025Renormalizable Spectral-Shell Dynamics as the Origin of Neural Scaling Lawscited by 0
  34. 2025When Does Learning Renormalize? Sufficient Conditions for Power Law Spectral Dynamicscited by 0
  35. 2026Predicting Large Model Test Losses with a Noisy Quadratic Systemcited by 0
  36. 2026Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networkscited by 0
  37. 2026How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Lawscited by 0
  38. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0

50 most cited of 56 All 56 (JSON)