mlsanityData to 14 Oct 2026

Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained models

arXiv 2503.07325 ↗First posted 10 Mar 2025cs.LG stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Understanding and certifying the behavior of modern deep neural networks remains a fundamental challenge in reliable machine learning. We introduce a new class of data-dependent generalization bounds that apply directly to trained models, without any modification. In particular, we present an exactly computable bound that is non-vacuous across all evaluated networks, including ImageNet-scale models with 600M parameters. This this is the first work showing that meaningful generalization guarantees are achievable even for large, unaltered deep networks. Our approach reveals that generalization is governed by the interaction between the trained model and the geometry of the data distribution. We decompose the generalization error into two interpretable components: a distributional complexity term, capturing how the data mass is distributed across the input space, and local model-behavior terms, capturing the network's behavior within individual regions. This joint dependence identifies where and why generalization gaps arise. Empirically, some components of our bound are highly predictive of the true test error, and the bound tightens when the partition aligns with the intrinsic data geometry, highlighting data-dependent local regularity as a key driver of generalization.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

37 of 61 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  11. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  12. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
25 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  2. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  3. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 126
  4. 2020Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descentcited by 119
  5. 2016Robust Large Margin Deep Neural Networkscited by 118
  6. 2020Tighter risk certificates for neural networkscited by 112
  7. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  8. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 98
  9. 2017Data-Dependent Stability of Stochastic Gradient Descentcited by 92
  10. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  11. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  12. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  13. 2022PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingalescited by 24
  14. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  15. 2022Non-Vacuous Generalisation Bounds for Shallow Neural Networkscited by 22
  16. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  17. 2021A General Framework for the Practical Disintegration of PAC-Bayesian Boundscited by 21
  18. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  19. 2022Neural Tangent Kernel Analysis of Deep Narrow Neural Networkscited by 11
  20. 2023Understanding prompt engineering may not require rethinking generalizationcited by 10
  21. 2022On Measuring Excess Capacity in Neural Networkscited by 9
  22. 2022Generalisation under gradient descent via deterministic PAC-Bayescited by 9
  23. 2022Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficultycited by 9
  24. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  25. 2024Gentle Local Robustness implies Generalizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.