Three Mechanisms of Feature Learning in a Linear Network
Not on the map. Why?
Abstract (arXiv)
Understanding the dynamics of neural networks in different width regimes is crucial for improving their training and performance. We present an exact solution for the learning dynamics of a one-hidden-layer linear network, with one-dimensional data, across any finite width, uniquely exhibiting both kernel and feature learning phases. This study marks a technical advancement by enabling the analysis of the training trajectory from any initialization and a detailed phase diagram under varying common hyperparameters such as width, layer-wise learning rates, and scales of output and initialization. We identify three novel prototype mechanisms specific to the feature learning regime: (1) learning by alignment, (2) learning by disalignment, and (3) learning by rescaling, which contrast starkly with the dynamics observed in the kernel regime. Our theoretical findings are substantiated with empirical evidence showing that these mechanisms also manifest in deep nonlinear networks handling real-world tasks, enhancing our understanding of neural network training dynamics and guiding the design of more effective learning strategies.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 13 Jan 2024.
Builds on
49 of 65 references.
References · 49
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
37 more
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2018Training behavior of deep neural network in frequency domaincited by 174
- 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
- 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
- 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
- 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
- 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
- 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
- 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
- 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
- 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
- 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
- 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
- 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
- 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
- 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
- 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
- 2023Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limitcited by 35
- 2019A type of generalization error induced by initialization in deep neural networkscited by 33
- 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
- 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
- 2021Properties of the After Kernelcited by 27
- 2022Exact Solutions of a Deep Linear Networkcited by 24
- 2021Convergence of gradient descent for learning linear neural networkscited by 22
- 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
- 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
- 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
- 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
- 2020Asymptotics of Wide Convolutional Neural Networkscited by 15
- 2020An analytic theory of shallow networks dynamics for hinge loss classificationcited by 14
- 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
- 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
- 2021Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field trainingcited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.