Approximation Results for Gradient Descent trained Neural Networks
Not on the map. Why?
Abstract (arXiv)
The paper contains approximation guarantees for neural networks that are trained with gradient flow, with error measured in the continuous L2(mathbbSd-1)-norm on the d-dimensional unit sphere and targets that are Sobolev smooth. The networks are fully connected of constant depth and increasing width. Although all layers are trained, the gradient flow convergence is based on a neural tangent kernel (NTK) argument for the non-convex second but last layer. Unlike standard NTK analysis, the continuous error norm implies an under-parametrized regime, possible by the natural smoothness assumption required for approximation. The typical over-parametrization re-enters the results in form of a loss in approximation rate relative to established approximation methods for Sobolev smooth functions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 57 of its 76 references that are in the map. First posted on arXiv 9 Sep 2023.
Builds on
57 of 76 references.
References · 57
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2016Error bounds for approximations with deep ReLU networkscited by 567
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
- 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
- 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
38 more
- 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
- 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
- 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
- 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
- 2020Deep Network Approximation for Smooth Functionscited by 163
- 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
- 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
- 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
- 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
- 2019Deep Neural Network Approximation Theorycited by 92
- 2019The phase diagram of approximation rates for deep neural networkscited by 86
- 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
- 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
- 2019Scaling description of generalization with number of parameters in deep learningcited by 83
- 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
- 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
- 2019A Theoretical Analysis of Deep Neural Networks and Parametric PDEscited by 74
- 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
- 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
- 2021The Modern Mathematics of Deep Learningcited by 63
- 2015Provable approximation properties for deep neural networkscited by 60
- 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
- 2020The gap between theory and practice in function approximation with deep neural networkscited by 53
- 2019Approximation spaces of deep neural networkscited by 51
- 2019Nonlinear Approximation via Compositionscited by 37
- 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
- 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
- 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
- 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
- 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
- 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
- 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
- 2020Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularitiescited by 20
- 2020Complexity Measures for Neural Networks with General Activation Functions Using Path-based Normscited by 19
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.