mlsanityData to 14 Oct 2026

Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels

arXiv 2103.01210 ↗First posted 1 Mar 2021cs.LG stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

We study the relative power of learning with gradient descent on differentiable models, such as neural networks, versus using the corresponding tangent kernels. We show that under certain conditions, gradient descent achieves small error only if a related tangent kernel method achieves a non-trivial advantage over random guessing (a.k.a. weak learning), though this advantage might be very small even when gradient descent can achieve arbitrarily high accuracy. Complementing this, we show that without these conditions, gradient descent can in fact learn with small error even when no kernel method, in particular using the tangent kernel, can achieve a non-trivial advantage over random guessing.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 21 of its 47 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

21 of 47 references.

References · 21

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  10. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  11. 2019Linearized two-layers neural networks in high dimensioncited by 155
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
9 more
  1. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  2. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  3. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  4. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  5. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  6. 2020Learning Parities with Neural Networkscited by 78
  7. 2017Learning ReLUs via Gradient Descentcited by 54
  8. 2020Approximate is Good Enough: Probabilistic Variants of Dimensional and Margin Complexitycited by 25
  9. 2020Poly-time universality and limitations of deep learningcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 13; 2023: 10; 2024: 4; 2025: 3; 2026: 3.

41 papers.

Sort:

By area:

  1. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  2. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  3. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  4. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  5. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  6. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  7. 2023Transformers learn through gradual rank increasecited by 52
  8. 2021What can linearized neural networks actually say about generalization?cited by 49
  9. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  10. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  11. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  12. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
29 more
  1. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  2. 2021On the Power of Differentiable Learning versus PAC and SQ Learningcited by 28
  3. 2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
  4. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  5. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 21
  6. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  7. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  8. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
  9. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  10. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  11. 2024On the Complexity of Learning Sparse Functions with Statistical and Gradient Queriescited by 15
  12. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14
  13. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  14. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  15. 2022An initial alignment between neural network and target is needed for gradient descent to learncited by 12
  16. 2021On Energy-Based Models with Overparametrized Shallow Neural Networkscited by 8
  17. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  18. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  19. 2022Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficultycited by 5
  20. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  21. 2023Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learningcited by 3
  22. 2026Positive Distribution Shift as a Framework for Understanding Tractable Learningcited by 3
  23. 2022Long-Tailed Learning Requires Feature Learningcited by 1
  24. 2023Tight conditions for when the NTK approximation is validcited by 1
  25. 2024RedEx: Beyond Fixed Representation Methods via Convex Optimizationcited by 1
  26. 2025Feature learning is decoupled from generalization in high capacity neural networkscited by 1
  27. 2026The Benefits of Temporal Correlations: SGD Learns k-Juntas from Random Walks Efficientlycited by 1
  28. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  29. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0