mlsanityData to 14 Oct 2026

Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

arXiv 2112.15383 ↗First posted 31 Dec 2021stat.ML cs.LG physics.data-ancited by 53

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) are powerful tools for compressing and distilling information. Their scale and complexity, often involving billions of inter-dependent parameters, render direct microscopic analysis difficult. Under such circumstances, a common strategy is to identify slow variables that average the erratic behavior of the fast microscopic variables. Here, we identify a similar separation of scales occurring in fully trained finitely over-parameterized deep convolutional neural networks (CNNs) and fully connected networks (FCNs). Specifically, we show that DNN layers couple only through the second moment (kernels) of their activations and pre-activations. Moreover, the latter fluctuates in a nearly Gaussian manner. For infinite width DNNs, these kernels are inert, while for finite ones they adapt to the data and yield a tractable data-aware Gaussian Process. The resulting thermodynamic theory of deep learning yields accurate predictions in various settings. In addition, it provides new ways of analyzing and understanding DNNs in general.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

42 of 73 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2018On Lazy Training in Differentiable Programmingcited by 675
  6. 2017Deep Neural Networks as Gaussian Processescited by 622
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  8. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  11. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  12. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
30 more
  1. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  2. 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 227
  3. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 197
  8. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  9. 2021The Principles of Deep Learning Theorycited by 173
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  12. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  13. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  14. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  15. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  16. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  17. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  18. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  19. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  20. 2020Neural Kernels Without Tangentscited by 56
  21. 2020Is SGD a Bayesian sampler? Well, almostcited by 50
  22. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  23. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  24. 2019Improved Bounds for Discretization of Langevin Diffusions: Near-Optimal Rates without Convexitycited by 39
  25. 2020Deep kernel processescited by 34
  26. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  27. 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
  28. 2019Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspectivecited by 11
  29. 2021Lower Bounds on the Generalization Error of Nonlinear Learning Modelscited by 6
  30. 2020Normalization effects on shallow neural networks and related asymptotic expansionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 15; 2024: 13; 2025: 13; 2026: 8.

53 papers.

Sort:

By area:

  1. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  3. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  4. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  5. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41
  6. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  7. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  8. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  9. 2023Neural Network Field Theories: Non-Gaussianity, Actions, and Localitycited by 22
  10. 2022Random matrix analysis of deep neural network weight matricescited by 21
  11. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  12. 2023Quantitative CLTs in Deep Neural Networkscited by 20
38 more
  1. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  2. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  3. 2023Structures of Neural Network Effective Theoriescited by 13
  4. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 13
  5. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  6. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  7. 2023Finite-time Lyapunov exponents of deep neural networkscited by 12
  8. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  9. 2024High-dimensional learning of narrow neural networkscited by 12
  10. 2024Coding schemes in neural networks learning classification taskscited by 11
  11. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  12. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  13. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  14. 2024Wilsonian Renormalization of Neural Network Gaussian Processescited by 8
  15. 2024Graph Neural Networks Do Not Always Oversmoothcited by 7
  16. 2023Field theory for optimal signal propagation in ResNetscited by 6
  17. 2024Random features and polynomial rulescited by 6
  18. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  19. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 4
  20. 2024Can Kernel Methods Explain How the Data Affects Neural Collapse?cited by 4
  21. 2025Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexitycited by 4
  22. 2023Convolutional Deep Kernel Machinescited by 3
  23. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 3
  24. 2024Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matricescited by 2
  25. 2025Grokking vs. Learning: Same Features, Different Encodingscited by 2
  26. 2025GrokAlign: Geometric Characterisation and Acceleration of Grokkingcited by 2
  27. 2026A unified theory of feature learning in RNNs and DNNscited by 2
  28. 2026Statistical Properties of Training & Generalizationcited by 2
  29. 2023Speed Limits for Deep Learningcited by 1
  30. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 1
  31. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  32. 2026Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformerscited by 1
  33. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  34. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  35. 2023Spectral-Bias and Kernel-Task Alignment in Physically Informed Neural Networkscited by 0
  36. 2024Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machinescited by 0
  37. 2025Deep Neural Nets as Hamiltonianscited by 0
  38. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0

50 most cited of 53 All 53 (JSON)