mlsanityData to 7 Oct 2026

Approximation results for Gradient Descent trained Shallow Neural Networks in 1d

arXiv 2209.08399 ↗First posted 17 Sep 2022cs.LG cs.NA math.NAcited by 7

Not on the map. Why?

Abstract (arXiv)

Two aspects of neural networks that have been extensively studied in the recent literature are their function approximation properties and their training by gradient descent methods. The approximation problem seeks accurate approximations with a minimal number of weights. In most of the current literature these weights are fully or partially hand-crafted, showing the capabilities of neural networks but not necessarily their practical performance. In contrast, optimization theory for neural networks heavily relies on an abundance of weights in over-parametrized regimes. This paper balances these two demands and provides an approximation result for shallow networks in 1d with non-convex weight optimization by gradient descent. We consider finite width networks and infinite sample limits, which is the typical setup in approximation theory. Technically, this problem is not over-parametrized, however, some form of redundancy reappears as a loss in approximation rate compared to best possible rates.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 55 of its 74 references that are in the map. First posted on arXiv 17 Sep 2022.

Builds on

55 of 74 references.

References · 55

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2016Error bounds for approximations with deep ReLU networkscited by 567
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  12. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
38 more
  1. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  2. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  3. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  4. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  5. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  6. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  7. 2020Deep Network Approximation for Smooth Functionscited by 163
  8. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  10. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  11. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  12. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  13. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  14. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  15. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  16. 2019Deep Neural Network Approximation Theorycited by 92
  17. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  18. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  19. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  20. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  21. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  22. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  23. 2019A Theoretical Analysis of Deep Neural Networks and Parametric PDEscited by 74
  24. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  25. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  26. 2021The Modern Mathematics of Deep Learningcited by 63
  27. 2015Provable approximation properties for deep neural networkscited by 60
  28. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  29. 2020The gap between theory and practice in function approximation with deep neural networkscited by 53
  30. 2019Approximation spaces of deep neural networkscited by 51
  31. 2019Nonlinear Approximation via Compositionscited by 37
  32. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
  33. 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
  34. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  35. 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
  36. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  37. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  38. 2020Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularitiescited by 20

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 2.

7 papers.

  1. 2024Convergence rates for the Adam optimizercited by 14
  2. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  3. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  4. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  5. 2025A brief review of the Deep BSDE method for solving high-dimensional partial differential equationscited by 5
  6. 2023Approximation Results for Gradient Descent trained Neural Networkscited by 2
  7. 2024Approximation and Gradient Descent Training with Neural Networkscited by 1