mlsanityData to 14 Oct 2026

A Generalization Bound for Nearly-Linear Networks

arXiv 2407.06765 ↗First posted 9 Jul 2024cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider nonlinear networks as perturbations of linear ones. Based on this approach, we present novel generalization bounds that become non-vacuous for networks that are close to being linear. The main advantage over the previous works which propose non-vacuous generalization bounds is that our bounds are a-priori: performing the actual training is not required for evaluating the bounds. To the best of our knowledge, they are the first non-vacuous generalization bounds for neural nets possessing this property.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

20 of 32 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  10. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  11. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  12. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 126
8 more
  1. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  2. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 108
  3. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  4. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  5. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  6. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  7. 2022Non-Vacuous Generalisation Bounds for Shallow Neural Networkscited by 22
  8. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18

Built on by

Not cited yet.