mlsanityData to 7 Oct 2026

Stronger generalization bounds for deep nets via a compression approach

arXiv 1802.05296 ↗First posted 14 Feb 2018cs.LGcited by 306

Not on the map. Why?

Abstract (arXiv)

Deep nets generalize well despite having more parameters than the number of training samples. Recent works try to give an explanation using PAC-Bayes and Margin-based analyses, but do not as yet result in sample complexity bounds better than naive parameter counting. The current paper shows generalization bounds that're orders of magnitude better in practice. These rely upon new succinct reparametrizations of the trained net --- a compression that is explicit and efficient. These yield generalization bounds via a simple compression-based framework introduced here. Our results also provide some theoretical justification for widespread empirical success in compressing deep nets. Analysis of correctness of our compression relies upon some newly identified textquotedblleft noise stabilitytextquotedblright properties of trained deep nets, which are also experimentally verified. The study of these properties and resulting generalization bounds are also extended to convolutional nets, which had eluded earlier attempts on proving generalization.

Key facts. Cited by 306 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 72; 2022: 69; 2023: 54; 2024: 32; 2025: 46; 2026: 32.

306 papers.

Sort:

By area:

  1. 2021The Values Encoded in Machine Learning Researchcited by 172
  2. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  3. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  4. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  5. 2022Sharpness-Aware Training for Freecited by 95
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  7. 2023Deep Model Fusion: A Surveycited by 89
  8. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  9. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  10. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  11. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  12. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
38 more
  1. 2022Error estimates for physics informed neural networks approximating the Navier-Stokes equationscited by 63
  2. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  3. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  4. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  5. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  6. 2021Exploring Memorization in Adversarial Trainingcited by 55
  7. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  8. 2023Empirical Design in Reinforcement Learningcited by 54
  9. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  10. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  11. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  12. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  13. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  14. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  15. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  16. 2023Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusioncited by 39
  17. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  18. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  19. 2021Stability of SGD: Tightness Analysis and Improved Boundscited by 33
  20. 2021Generalization bounds via distillationcited by 33
  21. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  22. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  23. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  24. 2021Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stabilitycited by 29
  25. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  26. 2021Robust Predictable Controlcited by 29
  27. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  28. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  29. 2023AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNscited by 26
  30. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  31. 2023Bridging Information-Theoretic and Geometric Compression in Language Modelscited by 26
  32. 2021Scaling Laws for Deep Learningcited by 25
  33. 2022On the Generalization Mystery in Deep Learningcited by 25
  34. 2023Pruning Deep Neural Networks from a Sparsity Perspectivecited by 25
  35. 2021Measuring Generalization with Optimal Transportcited by 24
  36. 2022Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guaranteescited by 24
  37. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  38. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23

50 most cited of 306 All 306 (JSON)