mlsanityData to 14 Oct 2026

Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle

arXiv 2310.17378 ↗First posted 26 Oct 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in deep learning have given us some very promising results on the generalization ability of deep neural networks, however literature still lacks a comprehensive theory explaining why heavily over-parametrized models are able to generalize well while fitting the training data. In this paper we propose a PAC type bound on the generalization error of feedforward ReLU networks via estimating the Rademacher complexity of the set of networks available from an initial parameter vector via gradient descent. The key idea is to bound the sensitivity of the network's gradient to perturbation of the input data along the optimization trajectory. The obtained bound does not explicitly depend on the depth of the network. Our results are experimentally verified on the MNIST and CIFAR-10 datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

24 of 36 references.

References · 24

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2017Exploring Generalization in Deep Learningcited by 600
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  7. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  8. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  9. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  10. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  11. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
  12. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 120
12 more
  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  2. 2020A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networkscited by 85
  3. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  4. 2019Generalization bounds for deep convolutional neural networkscited by 59
  5. 2019PAC-Bayes with Backpropcited by 36
  6. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 31
  7. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  8. 2021Uniform Generalization Bounds for Overparameterized Neural Networkscited by 21
  9. 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
  10. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3
  11. 2021Gradient representations in ReLU networks as similarity functionscited by 1
  12. 2023Gradient Descent in Neural Networks as Sequential Learning in RKBScited by 1

Built on by

Not cited yet.