mlsanityData to 7 Oct 2026

Spring-block theory of feature learning in deep neural networks

arXiv 2407.19353 ↗First posted 28 Jul 2024cond-mat.dis-nn cond-mat.stat-mech cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Feature-learning deep nets progressively collapse data to a regular low-dimensional geometry. How this emerges from the collective action of nonlinearity, noise, learning rate, and other factors, has eluded first-principles theories built from microscopic neuronal dynamics. We exhibit a noise-nonlinearity phase diagram that identifies regimes where shallow or deep layers learn more effectively and propose a macroscopic mechanical theory that reproduces the diagram and links feature learning across layers to generalization.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 28 Jul 2024.

Builds on

51 of 74 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  9. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  10. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  11. 2017Three Factors Influencing Minima in SGDcited by 270
  12. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
38 more
  1. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  2. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  3. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  4. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  5. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  6. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  7. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  8. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  9. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  10. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  11. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  12. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  13. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 58
  14. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  15. 2014Training Convolutional Networks with Noisy Labelscited by 47
  16. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  17. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  18. 2022A Law of Data Separation in Deep Learningcited by 36
  19. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  20. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  21. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  22. 2020Separation and Concentration in Deep Networkscited by 28
  23. 2021Unveiling the structure of wide flat minima in neural networkscited by 27
  24. 2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 24
  25. 2022Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutionscited by 22
  26. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  27. 2023A Neural Collapse Perspective on Feature Evolution in Graph Neural Networkscited by 19
  28. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 19
  29. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  30. 2023A Rainbow in Deep Network Black Boxescited by 18
  31. 2024Early alignment in two-layer networks training is a two-edged swordcited by 17
  32. 2022Homophily modulates double descent generalization in graph convolution networkscited by 13
  33. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  34. 2024The boundary of neural network trainability is fractalcited by 12
  35. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  36. 2023Injectivity of ReLU networks: perspectives from statistical physicscited by 10
  37. 2018Geometry of energy landscapes and the optimizability of deep neural networkscited by 9
  38. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networkscited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024A Law of Next-Token Prediction in Large Language Modelscited by 15
  2. 2026Deep Neural Regression Collapsecited by 1