mlsanityData to 7 Oct 2026

PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization

arXiv 2211.13609 ↗First posted 24 Nov 2022cs.LG stat.MLcited by 75

Not on the map. Why?

Abstract (arXiv)

While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works. In this paper, we develop a compression approach based on quantizing neural network parameters in a linear subspace, profoundly improving on previous results to provide state-of-the-art generalization bounds on a variety of tasks, including transfer learning. We use these tight bounds to better understand the role of model size, equivariance, and the implicit biases of optimization, for generalization in deep learning. Notably, we find large models can be compressed to a much greater extent than previously known, encapsulating Occam's razor. We also argue for data-independent bounds in explaining generalization.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 44 of its 77 references that are in the map. First posted on arXiv 24 Nov 2022.

Builds on

44 of 77 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Wide Residual Networkscited by 3,057
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2016Group Equivariant Convolutional Networkscited by 1,154
  10. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
32 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2018Spherical CNNscited by 452
  3. 2019General E(2)-Equivariant Steerable CNNscited by 441
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  7. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  9. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  10. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  12. 2020Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Datacited by 248
  13. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  14. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  15. 2014Qualitatively characterizing neural network optimization problemscited by 186
  16. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  17. 2020Tighter risk certificates for neural networkscited by 112
  18. 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 90
  19. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  20. 2020On the Benefits of Invariance in Neural Networkscited by 78
  21. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  22. 2014Fastfood: Approximate Kernel Expansions in Loglinear Timecited by 72
  23. 2020On the role of data in PAC-Bayes boundscited by 64
  24. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  25. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  26. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  27. 2016Towards the Limit of Network Quantizationcited by 49
  28. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  29. 2019PAC-Bayes with Backpropcited by 36
  30. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  31. 2021Understanding the Generalization Benefit of Model Invariance from a Data Perspectivecited by 22
  32. 2021Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 10; 2025: 21; 2026: 20.

75 papers.

Sort:

By area:

  1. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  2. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  3. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  4. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  5. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  6. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
  7. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  8. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  9. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  10. 2023More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validitycited by 17
  11. 2023Learning via Wasserstein-Based High Probability Generalisation Boundscited by 11
  12. 2023Understanding prompt engineering may not require rethinking generalizationcited by 10
38 more
  1. 2024Efficient Sketches for Training Data Attribution and Studying the Loss Landscapecited by 10
  2. 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
  3. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  4. 2023Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!cited by 8
  5. 2024Multitask methods for predicting molecular properties from heterogeneous datacited by 8
  6. 2025When Do Neural Networks Learn World Models?cited by 8
  7. 2023Norm-based Generalization Bounds for Compositionally Sparse Neural Networkscited by 7
  8. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  9. 2023PAC-Bayes Generalization Certificates for Learned Inductive Conformal Predictioncited by 6
  10. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  11. 2023Comparing Comparators in Generalization Boundscited by 5
  12. 2025From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learningcited by 5
  13. 2025Compute-Optimal LLMs Provably Generalize Better With Scalecited by 5
  14. 2023Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networkscited by 4
  15. 2023Input margins can predict generalization toocited by 4
  16. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  17. 2023Perspectives on the State and Future of Deep Learning - 2023cited by 4
  18. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  19. 2025How good is PAC-Bayes at explaining generalisation?cited by 4
  20. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  21. 2026Towards A Unified PAC-Bayesian Framework for Norm-based Generalization Boundscited by 4
  22. 2023The fine print on tempered posteriorscited by 3
  23. 2024Just How Flexible are Neural Networks in Practice?cited by 3
  24. 2024Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworkscited by 3
  25. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  26. 2026FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustnesscited by 3
  27. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  28. 2025Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained modelscited by 2
  29. 2025Fast Rate Bounds for Multi-Task and Meta-Learning with Different Sample Sizescited by 2
  30. 2025Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processescited by 2
  31. 2025Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifiercited by 2
  32. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  33. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  34. 2026Neural Weight Norm = Kolmogorov Complexitycited by 2
  35. 2023Estimating optimal PAC-Bayes bounds with Hamiltonian Monte Carlocited by 1
  36. 2023A note on regularised NTK dynamics with an application to PAC-Bayesian trainingcited by 1
  37. 2024A practical generalization metric for deep networks benchmarkingcited by 1
  38. 2025Optimizing Data Augmentation through Bayesian Model Selectioncited by 1

50 most cited of 75 All 75 (JSON)