Estimating the Generalization in Deep Neural Networks via Sparsity
Not on the map. Why?
Abstract (arXiv)
Generalization is the key capability for deep neural networks (DNNs). However, it is challenging to give a reliable measure of the generalization ability of a DNN via only its nature. In this paper, we propose a novel method for estimating the generalization gap based on network sparsity. In our method, two key quantities are proposed first. They have close relationship with the generalization ability and can be calculated directly from the training results alone. Then a simple linear model involving two key quantities are constructed to give accurate estimation of the generalization gap. By training DNNs with a wide range of generalization gap on popular datasets, we show that our key quantities and linear model could be efficient tools for estimating the generalization gap of DNNs.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 2 Apr 2021.
Builds on
30 of 43 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
- 2017Spectrally-normalized margin bounds for neural networkscited by 681
- 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
- 2017Exploring Generalization in Deep Learningcited by 595
18 more
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
- 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
- 2015Norm-Based Capacity Control in Neural Networkscited by 299
- 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
- 2016Robust Large Margin Deep Neural Networkscited by 117
- 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
- 2018Large Margin Deep Networks for Classificationcited by 110
- 2018On the importance of single directions for generalizationcited by 99
- 2016Group Sparse Regularization for Deep Neural Networkscited by 89
- 2019The Generalization-Stability Tradeoff In Neural Network Pruningcited by 54
- 2018Revisiting the Importance of Individual Units in CNNs via Ablationcited by 32
- 2020Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNscited by 15
- 2017Fast Rates for Empirical Risk Minimization of Strict Saddle Problemscited by 7
- 2019On improving deep learning generalization with adaptive sparse connectivitycited by 7
- 2021Quantitative Performance Assessment of CNN Units via Topological Entropy Calculationcited by 6
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.
2 papers.