mlsanityData to 7 Oct 2026

Sparsity-aware generalization theory for deep neural networks

arXiv 2307.00426 ↗First posted 1 Jul 2023cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Deep artificial neural networks achieve surprising generalization abilities that remain poorly understood. In this paper, we present a new approach to analyzing generalization for deep feed-forward ReLU networks that takes advantage of the degree of sparsity that is achieved in the hidden layer activations. By developing a framework that accounts for this reduced effective model size for each input sample, we are able to show fundamental trade-offs between sparsity and generalization. Importantly, our results make no strong assumptions about the degree of sparsity achieved by the model, and it improves over recent norm-based approaches. We illustrate our results numerically, demonstrating non-vacuous bounds when coupled with data-dependent priors in specific settings, even in over-parametrized models.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 1 Jul 2023.

Builds on

27 of 36 references.

References · 27

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  6. 2017Exploring Generalization in Deep Learningcited by 595
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  9. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  10. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  11. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  12. 2015Norm-Based Capacity Control in Neural Networkscited by 299
15 more
  1. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  2. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  3. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  4. 2019A Primer on PAC-Bayesian Learningcited by 130
  5. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  6. 2017Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priorscited by 81
  7. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  8. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  9. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  10. 2020Generalization bounds for deep learningcited by 38
  11. 2018On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyondcited by 24
  12. 2020Adversarial Robustness of Supervised Sparse Codingcited by 22
  13. 2022Adversarial robustness of sparse local Lipschitz predictorscited by 11
  14. 2020De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictorscited by 7
  15. 2023Norm-based Generalization Bounds for Compositionally Sparse Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 3; 2026: 3.

8 papers.

  1. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  2. 2023Size Lowerbounds for Deep Operator Networkscited by 6
  3. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  4. 2023Investigating the Ability of PINNs To Solve Burgers' PDE Near Finite-Time BlowUpcited by 2
  5. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  6. 2025LightSNN: Lightweight Architecture Search for Sparse and Accurate Spiking Neural Networkscited by 1
  7. 2026How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNscited by 1
  8. 2026Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networkscited by 0