mlsanityData to 13 Oct 2026

What can linearized neural networks actually say about generalization?

arXiv 2106.06770 ↗First posted 12 Jun 2021cs.LG cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

For certain infinitely-wide neural networks, the neural tangent kernel (NTK) theory fully characterizes generalization, but for the networks used in practice, the empirical NTK only provides a rough first-order approximation. Still, a growing body of work keeps leveraging this approximation to successfully analyze important deep learning phenomena and design algorithms for new applications. In our work, we provide strong empirical evidence to determine the practical validity of such approximation by conducting a systematic comparison of the behavior of different neural networks and their linear approximations on different tasks. We show that the linear approximations can indeed rank the learning complexity of certain tasks for neural networks, even when they achieve very different performances. However, in contrast to what was previously reported, we discover that neural networks do not always perform better than their kernel approximations, and reveal that the performance gap heavily depends on architecture, dataset size and training task. We discover that networks overfit to these tasks mostly due to the evolution of their kernel during training, thus, revealing a new type of implicit bias.

Key facts. Cited by 49 papers in the map (data through 13 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 12 Jun 2021.

Builds on

32 of 46 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  5. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,183
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,666
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 737
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 652
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 573
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  11. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 306
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 263
20 more
  1. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  2. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  3. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  4. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 172
  5. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  6. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  7. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  8. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  9. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  10. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  11. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 58
  12. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  13. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  14. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  15. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  16. 2020Geometric compression of invariant manifolds in neural netscited by 28
  17. 2020Predicting Training Time Without Trainingcited by 22
  18. 2020Neural Anisotropy Directionscited by 17
  19. 2021A Unified Paths Perspective for Pruning at Initializationcited by 14
  20. 2021Uniform Convergence, Adversarial Spheres and a Simple Remedycited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 12; 2023: 10; 2024: 11; 2025: 11; 2026: 3.

49 papers.

Sort:

By area:

  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 219
  2. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  3. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  4. 2024Improved Implicit Neural Representation with Fourier Reparameterized Trainingcited by 36
  5. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  6. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  7. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  8. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  9. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  10. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  11. 2022What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness?cited by 21
  12. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
37 more
  1. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  2. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  3. 2022Can we achieve robustness from data alone?cited by 16
  4. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  5. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  6. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  7. 2026A Defense of the Quadratic Modelcited by 11
  8. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 10
  9. 2024Neural Lineagecited by 9
  10. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  11. 2021Representation Alignment in Neural Networkscited by 6
  12. 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
  13. 2022Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficultycited by 5
  14. 2023Towards a Phenomenological Understanding of Neural Networks: Datacited by 4
  15. 2024Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusioncited by 4
  16. 2025Feature Identification via the Empirical NTKcited by 4
  17. 2025Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networkscited by 3
  18. 2022A Statistical Model for Predicting Generalization in Few-Shot Classificationcited by 2
  19. 2024SGD with memory: fundamental properties and stochastic accelerationcited by 2
  20. 2024Geometric Inductive Biases of Deep Networks: The Role of Data and Architecturecited by 2
  21. 2025Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lenscited by 2
  22. 2025Toward a Holistic Approach to Continual Model Mergingcited by 2
  23. 2022Label Alignment Regularization for Distribution Shiftcited by 1
  24. 2025Cognitive Activation and Chaotic Dynamics in Large Language Models: A Quasi-Lyapunov Analysis of Reasoning Mechanismscited by 1
  25. 2025Laplace Sample Information: Data Informativeness Through a Bayesian Lenscited by 1
  26. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1
  27. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  28. 2026Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tailcited by 1
  29. 2023Kernels, Data & Physicscited by 0
  30. 2023A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networkscited by 0
  31. 2023Modify Training Directions in Function Space to Reduce Generalization Errorcited by 0
  32. 2023Rethinking Adversarial Training with Neural Tangent Kernelcited by 0
  33. 2024Dropout MPC: An Ensemble Neural MPC Approach for Systems with Learned Dynamicscited by 0
  34. 2024ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networkscited by 0
  35. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  36. 2025FLAIR: Frequency- and Locality-Aware Implicit Neural Representationscited by 0
  37. 2026Target noise: A pre-training based neural network initialization for efficient high resolution learningcited by 0