Online Learning for the Random Feature Model in the Student-Teacher Framework
Not on the map. Why?
Abstract (arXiv)
Deep neural networks are widely used prediction algorithms whose performance often improves as the number of weights increases, leading to over-parametrization. We consider a two-layered neural network whose first layer is frozen while the last layer is trainable, known as the random feature model. We study over-parametrization in the context of a student-teacher framework by deriving a set of differential equations for the learning dynamics. For any finite ratio of hidden layer size and input dimension, the student cannot generalize perfectly, and we compute the non-zero asymptotic generalization error. Only when the student's hidden layer size is exponentially larger than the input dimension, an approach to perfect generalization is possible.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 69 references that are in the map. First posted on arXiv 24 Mar 2023.
Builds on
28 of 69 references.
References · 28
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
16 more
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
- 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
- 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
- 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
- 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
- 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
- 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
- 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
- 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
- 2020Analysis of Stochastic Gradient Descent in Continuous Timecited by 24
- 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 22
- 2019Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activationcited by 14
- 2021Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machinescited by 7
Built on by
Not cited yet.