mlsanityData to 7 Oct 2026

Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

arXiv 1909.11274 ↗First posted 25 Sep 2019cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches. To resolve this issue, several attempts have been made. Among them, the compression based bound is one of the promising approaches. However, the compression based bound can be applied only to a compressed network, and it is not applicable to the non-compressed original network. In this paper, we give a unified frame-work that can convert compression based bounds to those for non-compressed original networks. The bound gives even better rate than the one for the compressed network by improving the bias term. By establishing the unified frame-work, we can obtain a data dependent generalization error bound which gives a tighter evaluation than the data independent ones.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 9; 2024: 3; 2025: 9; 2026: 3.

39 papers.

Sort:

  1. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  2. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  3. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  4. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  5. 2021Generalization bounds via distillationcited by 33
  6. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  7. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  8. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19
  9. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  10. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  11. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  12. 2022On Measuring Excess Capacity in Neural Networkscited by 9
27 more
  1. 2023Koopman-based generalization bound: New aspect for full-rank weightscited by 9
  2. 2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
  3. 2023Deep Learning with Kernels through RKHM and the Perron-Frobenius Operatorcited by 7
  4. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  5. 2022On generalization bounds for deep networks based on loss surface implicit regularizationcited by 5
  6. 2023Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGDcited by 5
  7. 2023Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networkscited by 4
  8. 2023Generalization Guarantees via Algorithm-dependent Rademacher Complexitycited by 4
  9. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  10. 2025Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycited by 2
  11. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  12. 2025Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSscited by 1
  13. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  14. 2025Operator-Based Generalization Bound for Deep Learning: Insights on Multi-Task Learningcited by 1
  15. 2021Heating up decision boundaries: isocapacitory saturation, adversarial scenarios and generalization boundscited by 0
  16. 2022Approximability and Generalisationcited by 0
  17. 2023Proximity to Losslessly Compressible Parameterscited by 0
  18. 2023Distilling the Unknown to Unveil Certaintycited by 0
  19. 2023Scalable Federated Learning for Clients with Different Input Image Sizes and Numbers of Output Categoriescited by 0
  20. 2024Dimensionality-induced information loss of outliers in deep neural networkscited by 0
  21. 2025Why and When Deep is Better than Shallow: Implementation-Agnostic State-Transition Model of Deep Learningcited by 0
  22. 2025Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropycited by 0
  23. 2025Position: Many generalization measures for deep learning are fragilecited by 0
  24. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0
  25. 2025Block Coordinate Descent for Neural Networks Provably Finds Global Minimacited by 0
  26. 2026Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networkscited by 0
  27. 2026Unified generalization analysis for physics informed neural networkscited by 0