mlsanityData to 14 Oct 2026

Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent

arXiv 2606.06772 ↗First posted 4 Jun 2026stat.ML cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable success of deep learning. We establish quantitative bounds showing that kernel gradient descent in the reproducing kernel Hilbert space induced by the deterministic infinite-width neural tangent kernel approximates finite-width deep regression with smooth activations under gradient descent (GD) and stochastic gradient descent (SGD) training. The approximation gap is governed by the network width and training horizon, with an additional stochastic gradient error in the SGD case. This connection provides a general mechanism for transferring learning-theoretic guarantees from kernel methods to deep regression. As an application, under general source and effective dimension conditions, we show that both GD- and SGD-trained DNNs attain the minimax-optimal excess population risk rate, up to logarithmic factors, provided that the network width grows polynomially in the sample size. To the best of our knowledge, these are the first such guarantees for standard fully connected deep neural networks with smooth activations trained by GD and SGD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

21 of 32 references.

References · 21

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  7. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  8. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  9. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  10. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  11. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  12. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
9 more
  1. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  2. 2018Learning with SGD and Random Featurescited by 33
  3. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 26
  4. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  5. 2023How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descentcited by 10
  6. 2023Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networkscited by 9
  7. 2024Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methodscited by 7
  8. 2023Analysis of the expected L2 error of an over-parametrized deep neural network estimate learned by gradient descent without regularizationcited by 4
  9. 2024Generalization Analysis of Online Stochastic Gradient Descent for Overparameterized Two-Layer Neural Networkscited by 4

Built on by

Not cited yet.