mlsanityData to 14 Oct 2026

Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

arXiv 1910.01619 ↗First posted 3 Oct 2019cs.LG math.OC stat.MLcited by 79

Not on the map. Why?

Abstract (arXiv)

Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by he Neural Tangent Kernels (NTKs). NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized models, suggesting insufficiency of this theory. Towards closing this gap, we investigate the training of over-parametrized neural networks that are beyond the NTK regime yet still governed by the Taylor expansion of the network. We bring forward the idea of randomizing the neural networks, which allows them to escape their NTK and couple with quadratic models. We show that the optimization landscape of randomized two-layer networks are nice and amenable to escaping-saddle algorithms. We prove concrete generalization and expressivity results on these randomized networks, which lead to sample complexity bounds (of learning certain simple functions) that match the NTK and can in addition be better by a dimension factor when mild distributional assumptions are present. We demonstrate that our randomization technique can be generalized systematically beyond the quadratic case, by using it to find networks that are coupled with higher-order terms in their Taylor series.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 23; 2023: 20; 2024: 10; 2025: 4; 2026: 5.

79 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  5. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  6. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  7. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  8. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  9. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  10. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  11. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  12. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 40
38 more
  1. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 39
  2. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  3. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  4. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  5. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  6. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  7. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 26
  8. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  9. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  10. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 22
  11. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  12. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  13. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  14. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  15. 2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
  16. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 20
  17. 2023Six Lectures on Linearized Neural Networkscited by 20
  18. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  19. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
  20. 2021Optimization-Based Separations for Neural Networkscited by 17
  21. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  22. 2023Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networkscited by 16
  23. 2021A Probabilistic Approach to Neural Network Pruningcited by 15
  24. 2021Optimal Gradient-based Algorithms for Non-concave Bandit Optimizationcited by 15
  25. 2022Fighting Fire with Fire: Avoiding DNN Shortcuts through Primingcited by 15
  26. 2022Feature selection with gradient descent on two-layer networks in low-rotation regimescited by 14
  27. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 14
  28. 2026A Defense of the Quadratic Modelcited by 14
  29. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  30. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  31. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  32. 2022Non-convex online learning via algorithmic equivalencecited by 11
  33. 2021Sample Complexity and Overparameterization Bounds for Temporal Difference Learning with Neural Network Approximationcited by 10
  34. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  35. 2023Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Datacited by 10
  36. 2021Going Beyond Linear RL: Sample Efficient Neural Function Approximationcited by 9
  37. 2023Depth Separation with Multilayer Mean-Field Networkscited by 9
  38. 2023Learning Hierarchical Polynomials with Three-Layer Neural Networkscited by 9

50 most cited of 79 All 79 (JSON)