mlsanityData to 7 Oct 2026

Neural Networks as Kernel Learners: The Silent Alignment Effect

arXiv 2111.00034 ↗First posted 29 Oct 2021stat.ML cs.LGcited by 106

Not on the map. Why?

Abstract (arXiv)

Neural networks in the lazy training regime converge to kernel machines. Can neural networks in the rich feature learning regime learn a kernel machine with a data-dependent kernel? We demonstrate that this can indeed happen due to a phenomenon we term silent alignment, which requires that the tangent kernel of a network evolves in eigenstructure while small and before the loss appreciably decreases, and grows only in overall scale afterwards. We show that such an effect takes place in homogenous neural networks with small initialization and whitened data. We provide an analytical treatment of this effect in the linear network case. In general, we find that the kernel develops a low-rank contribution in the early phase of training, and then evolves in overall scale, yielding a function equivalent to a kernel regression solution with the final network's tangent kernel. The early spectral learning of the kernel depends on the depth. We also demonstrate that non-whitened data can weaken the silent alignment effect.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

28 of 48 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  11. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  12. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
16 more
  1. 2018To understand deep learning we need to understand kernel learningcited by 186
  2. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  3. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  4. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  5. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  6. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  7. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  8. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  9. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  10. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  11. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  12. 2020Every Model Learned by Gradient Descent Is Approximately a Kernel Machinecited by 53
  13. 2020Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticitycited by 18
  14. 2020On the asymptotics of wide networks with polynomial activationscited by 17
  15. 2020Asymptotics of Wide Convolutional Neural Networkscited by 15
  16. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 14; 2023: 16; 2024: 26; 2025: 24; 2026: 24.

106 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  3. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  4. 2023A Spectral Condition for Feature Learningcited by 101
  5. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  6. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  7. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  8. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 47
  9. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  10. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  11. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  12. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
38 more
  1. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  2. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  3. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  4. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
  5. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  6. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  7. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  8. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  9. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  10. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  11. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  12. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  13. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  14. 2023Neural (Tangent Kernel) Collapsecited by 18
  15. 2023A Rainbow in Deep Network Black Boxescited by 18
  16. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  17. 2025Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfercited by 17
  18. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  19. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  20. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  21. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  22. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  23. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  24. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  25. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  26. 2024High-dimensional learning of narrow neural networkscited by 12
  27. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  28. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  29. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  30. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  31. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 11
  32. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  33. 2022Nonparametric learning of kernels in nonlocal operatorscited by 10
  34. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 10
  35. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 10
  36. 2022Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditionscited by 9
  37. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
  38. 2021Depth induces scale-averaging in overparameterized linear Bayesian neural networkscited by 7

50 most cited of 106 All 106 (JSON)