mlsanityData to 7 Oct 2026

Approximation and Gradient Descent Training with Neural Networks

arXiv 2405.11696 ↗First posted 19 May 2024cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

It is well understood that neural networks with carefully hand-picked weights provide powerful function approximation and that they can be successfully trained in over-parametrized regimes. Since over-parametrization ensures zero training error, these two theories are not immediately compatible. Recent work uses the smoothness that is required for approximation results to extend a neural tangent kernel (NTK) optimization argument to an under-parametrized regime and show direct approximation bounds for networks trained by gradient flow. Since gradient flow is only an idealization of a practical method, this paper establishes analogous results for networks trained by gradient descent.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

43 of 60 references.

References · 43

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2016Error bounds for approximations with deep ReLU networkscited by 567
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  11. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  12. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
31 more
  1. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  2. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  3. 2020Deep Network Approximation for Smooth Functionscited by 163
  4. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  5. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  6. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  7. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  8. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  9. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  10. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  11. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  12. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  13. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  14. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  15. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  16. 2021The Modern Mathematics of Deep Learningcited by 63
  17. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  18. 2020The gap between theory and practice in function approximation with deep neural networkscited by 53
  19. 2019Approximation spaces of deep neural networkscited by 51
  20. 2019Nonlinear Approximation and (Deep) ReLU Networkscited by 45
  21. 2019Nonlinear Approximation via Compositionscited by 37
  22. 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
  23. 2019Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizescited by 29
  24. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  25. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  26. 2020Complexity Measures for Neural Networks with General Activation Functions Using Path-based Normscited by 19
  27. 2020Sharp Representation Theorems for ReLU Networks with Precise Dependence on Depthcited by 13
  28. 2022Neural Tangent Kernel Analysis of Deep Narrow Neural Networkscited by 11
  29. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  30. 2022Approximation results for Gradient Descent trained Shallow Neural Networks in 1dcited by 7
  31. 2022Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probabilitycited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Convergence rates for the Adam optimizercited by 14