Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
Not on the map. Why?
Abstract (arXiv)
We investigate the stationary (late-time) training regime of single- and two-layer underparameterized linear neural networks within the continuum limit of stochastic gradient descent (SGD) for synthetic Gaussian data. In the case of a single-layer network in the weakly underparameterized regime, the spectrum of the noise covariance matrix deviates notably from the Hessian, which can be attributed to the broken detailed balance of SGD dynamics. The weight fluctuations are in this case generally anisotropic, but effectively experience an isotropic loss. For an underparameterized two-layer network, we describe the stochastic dynamics of the weights in each layer and analyze the associated stationary covariances. We identify the inter-layer coupling as a distinct source of anisotropy for the weight fluctuations. In contrast to the single-layer case, the weight fluctuations are effectively subject to an anisotropic loss, the flatness of which is inversely related to the fluctuation variance. We thereby provide an analytical derivation of the recently observed inverse variance-flatness relation in a model of a deep linear neural network.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 80 of its 130 references that are in the map. First posted on arXiv 23 Nov 2023.
Builds on
80 of 130 references.
References · 80
- 2016Deep Learning with Differential Privacycited by 2,728
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
38 more
- 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2020Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Userscited by 338
- 2017Implicit Regularization in Matrix Factorizationcited by 322
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
- 2017Three Factors Influencing Minima in SGDcited by 273
- 2017High-dimensional dynamics of generalization error in neural networkscited by 241
- 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
- 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
- 2018A mathematical theory of semantic development in deep neural networkscited by 186
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2021The Principles of Deep Learning Theorycited by 173
- 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
- 2015A Complete Recipe for Stochastic Gradient MCMCcited by 146
- 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
- 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
- 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
- 2015Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networkscited by 121
- 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
- 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
- 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
- 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
- 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
- 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
- 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
- 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
- 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
- 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
- 2018Escaping Saddles with Stochastic Gradientscited by 66
- 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
- 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
- 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
- 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 57
- 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
- 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
- 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
50 most cited of 80 All 80 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.