mlsanityData to 7 Oct 2026

Feature learning is decoupled from generalization in high capacity neural networks

arXiv 2507.19680 ↗First posted 25 Jul 2025cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Neural networks outperform kernel methods, sometimes by orders of magnitude, e.g. on staircase functions. This advantage stems from the ability of neural networks to learn features, adapting their hidden representations to better capture the data. We introduce a concept we call feature quality to measure this performance improvement. We examine existing theories of feature learning and demonstrate empirically that they primarily assess the strength of feature learning, rather than the quality of the learned features themselves. Consequently, current theories of feature learning do not provide a sufficient foundation for developing theories of neural network generalization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 62 of its 72 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

62 of 72 references.

References · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2018On the Spectral Bias of Neural Networkscited by 1,321
  7. 2017A Closer Look at Memorization in Deep Networkscited by 936
  8. 2022Toy Models of Superpositioncited by 928
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2017Deep Neural Networks as Gaussian Processescited by 617
  12. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
38 more
  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  3. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  4. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  5. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  6. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  7. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  8. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  9. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  10. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  11. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  12. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  13. 2020Learning Parities with Neural Networkscited by 77
  14. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  15. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 71
  16. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  17. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  18. 2022Limitations of Neural Collapse for Understanding Generalization in Deep Learningcited by 63
  19. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  20. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  21. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  22. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  23. 2021What can linearized neural networks actually say about generalization?cited by 49
  24. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  25. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  26. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  27. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  28. 2024Mathematical Models of Computation in Superpositioncited by 34
  29. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  30. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  31. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  32. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  33. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  34. 2019Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigmcited by 20
  35. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  36. 2020Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learningcited by 18
  37. 2020Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTKcited by 18
  38. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Position: Many generalization measures for deep learning are fragilecited by 0