mlsanityData to 7 Oct 2026

Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets

arXiv 2111.14039 ↗First posted 28 Nov 2021cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the generalization performance of global minima for implementing empirical risk minimization (ERM) on over-parameterized deep ReLU nets. Using a novel deepening scheme for deep ReLU nets, we rigorously prove that there exist perfect global minima achieving almost optimal generalization error bounds for numerous types of data under mild conditions. Since over-parameterization is crucial to guarantee that the global minima of ERM on deep ReLU nets can be realized by the widely used stochastic gradient descent (SGD) algorithm, our results indeed fill a gap between optimization and generalization.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 53 of its 92 references that are in the map. First posted on arXiv 28 Nov 2021.

Builds on

53 of 92 references.

References · 53

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2016Error bounds for approximations with deep ReLU networkscited by 567
  10. 2019Benign Overfitting in Linear Regressioncited by 544
  11. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
38 more
  1. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  4. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  5. 2016On the Expressive Power of Deep Neural Networkscited by 302
  6. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  7. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  8. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  9. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  10. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  11. 2018Universality of Deep Convolutional Neural Networkscited by 176
  12. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  13. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  14. 2019Harmless interpolation of noisy data in regressioncited by 121
  15. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  16. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  17. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  18. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  19. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  20. 2015Provable approximation properties for deep neural networkscited by 60
  21. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  22. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  23. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  24. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  25. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  26. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  27. 2018Approximation beats concentration? An approximation view on inference with smooth radial kernelscited by 33
  28. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  29. 2020Failures of model-dependent generalization bounds for least-norm interpolationcited by 26
  30. 2019Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problemscited by 23
  31. 2018On the Benefit of Width for Neural Networks: Disappearance of Bad Basinscited by 16
  32. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
  33. 2019Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variationcited by 14
  34. 2020Depth Selection for Deep ReLU Nets in Feature Extraction and Generalizationcited by 13
  35. 2018Generalization and Expressivity for Deep Netscited by 11
  36. 2019Deep Neural Networks for Rotation-Invariance Approximation and Learningcited by 10
  37. 2019Realization of spatial sparseness by deep ReLU nets with massive datacited by 10
  38. 2019Realizing data features by deep netscited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Can overfitted deep neural networks in adversarial training generalize? -- An approximation viewpointcited by 1
  2. 2024Component-based Sketching for Deep ReLU Netscited by 1
  3. 2024Consistency for Large Neural Networks: Regression and Classificationcited by 0
  4. 2025Feature Qualification by Deep Nets: A Constructive Approachcited by 0