Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning
Not on the map. Why?
Abstract (arXiv)
These notes are based on a lecture delivered by NC on March 2021, as part of an advanced course in Princeton University on the mathematical understanding of deep learning. They present a theory (developed by NC, NR and collaborators) of linear neural networks -- a fundamental model in the study of optimization and generalization in deep learning. Practical applications born from the presented theory are also discussed. The theory is based on mathematical tools that are dynamical in nature. It showcases the potential of such tools to push the envelope of our understanding of optimization and generalization in deep learning. The text assumes familiarity with the basics of statistical learning theory. Exercises (without solutions) are included.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 54 of its 71 references that are in the map. First posted on arXiv 25 Aug 2024.
Builds on
54 of 71 references.
References · 54
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2017Implicit Regularization in Matrix Factorizationcited by 322
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
- 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
- 2017High-dimensional dynamics of generalization error in neural networkscited by 241
- 2016Deep Learning without Poor Local Minimacited by 234
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
- 2019Blind Super-Resolution Kernel Estimation using an Internal-GANcited by 175
38 more
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
- 2016Matrix Completion has No Spurious Local Minimumcited by 125
- 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
- 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
- 2016Identity Matters in Deep Learningcited by 109
- 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
- 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
- 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
- 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
- 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
- 2020Deep Polynomial Neural Networkscited by 63
- 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
- 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
- 2018ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networkscited by 53
- 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
- 2018Quantum Entanglement in Deep Learning Architecturescited by 50
- 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
- 2021Implicit Regularization in Tensor Factorizationcited by 44
- 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
- 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
- 2020Implicit Rank-Minimizing Autoencodercited by 35
- 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
- 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
- 2017Expressive power of recurrent neural networkscited by 30
- 2020DO-Conv: Depthwise Over-parameterized Convolutional Layercited by 28
- 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
- 2016Gradient Descent Converges to Minimizerscited by 25
- 2017Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Designcited by 24
- 2021Convergence of gradient descent for learning linear neural networkscited by 22
- 2018Step Size Matters in Deep Learningcited by 19
- 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
- 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
- 2017On the Long-Term Memory of Deep Recurrent Networkscited by 14
- 2017Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositionscited by 13
- 2022On the Ability of Graph Neural Networks to Model Interactions Between Verticescited by 12
- 2020On implicit regularization: Morse functions and applications to matrix factorizationcited by 11
- 2015Deep SimNetscited by 10
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.