mlsanityData to 7 Oct 2026

A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs

arXiv 2106.04110 ↗First posted 8 Jun 2021cs.LG cond-mat.stat-mech stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) in the infinite width/channel limit have received much attention recently, as they provide a clear analytical window to deep learning via mappings to Gaussian Processes (GPs). Despite its theoretical appeal, this viewpoint lacks a crucial ingredient of deep learning in finite DNNs, laying at the heart of their success -- feature learning. Here we consider DNNs trained with noisy gradient descent on a large training set and derive a self consistent Gaussian Process theory accounting for strong finite-DNN and feature learning effects. Applying this to a toy model of a two-layer linear convolutional neural network (CNN) shows good agreement with experiments. We further identify, both analytical and numerically, a sharp transition between a feature learning regime and a lazy learning regime in this model. Strong finite-DNN effects are also derived for a non-linear two-layer fully connected network. Our self consistent theory provides a rich and versatile analytical framework for studying feature learning and other non-lazy effects in finite DNNs.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

32 of 50 references.

References · 32

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2014How transferable are features in deep neural networks?cited by 1,216
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  12. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
20 more
  1. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  2. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  3. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  4. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  5. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  6. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  7. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  8. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  9. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  10. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  11. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  12. 2020The Recurrent Neural Tangent Kernelcited by 67
  13. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  14. 2020Neural Kernels Without Tangentscited by 56
  15. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  16. 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
  17. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  18. 2020Asymptotics of Wide Convolutional Neural Networkscited by 15
  19. 2013Feature Learning in Deep Neural Networks - Studies on Speech Recognition Taskscited by 14
  20. 2019Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspectivecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 8; 2024: 9; 2025: 7; 2026: 2.

33 papers.

Sort:

  1. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  3. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  4. 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
  5. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  6. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  7. 2022Contrasting random and learned features in deep Bayesian linear regressioncited by 24
  8. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  9. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  10. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  11. 2021A theory of representation learning gives a deep generalisation of kernel methodscited by 15
  12. 2023Structures of Neural Network Effective Theoriescited by 13
21 more
  1. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  2. 2022Globally Gated Deep Linear Networkscited by 12
  3. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  4. 2025From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learningcited by 10
  5. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  6. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  7. 2024Graph Neural Networks Do Not Always Oversmoothcited by 7
  8. 2023Field theory for optimal signal propagation in ResNetscited by 6
  9. 2024Random features and polynomial rulescited by 6
  10. 2024Restoring balance: principled under/oversampling of data for optimal classificationcited by 6
  11. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  12. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  13. 2023A theory of data variability in Neural Network Bayesian inferencecited by 2
  14. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  15. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  16. 2024Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machinescited by 0
  17. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  18. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  19. 2025Deep Neural Nets as Hamiltonianscited by 0
  20. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  21. 2026Width-Robust Learnability in Mean-Field Bayesian Neural Networkscited by 0