Searching for Minimal Optimal Neural Networks
Not on the map. Why?
Abstract (arXiv)
Large neural network models have high predictive power but may suffer from overfitting if the training set is not large enough. Therefore, it is desirable to select an appropriate size for neural networks. The destructive approach, which starts with a large architecture and then reduces the size using a Lasso-type penalty, has been used extensively for this task. Despite its popularity, there is no theoretical guarantee for this technique. Based on the notion of minimal neural networks, we posit a rigorous mathematical framework for studying the asymptotic theory of the destructive technique. We prove that Adaptive group Lasso is consistent and can reconstruct the correct number of hidden nodes of one-hidden-layer feedforward networks with high probability. To the best of our knowledge, this is the first theoretical result establishing for the destructive technique.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 9 of its 22 references that are in the map. First posted on arXiv 27 Sep 2021.
Builds on
9 of 22 references.
References · 9
- 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 122
- 2016Group Sparse Regularization for Deep Neural Networkscited by 89
- 2016Learning the Number of Neurons in Deep Networkscited by 69
- 2018Deep Neural Networks for Estimation and Inferencecited by 62
- 2017Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classificationcited by 39
- 2020Consistent Feature Selection for Analytic Deep Neural Networkscited by 27
- 2015Auto-Sizing Neural Networks: With Applications to n-gram Language Modelscited by 9
- 2019Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translationcited by 6
- 2020Consistent feature selection for neural networks via Adaptive Group Lassocited by 6
Built on by
Citing papers per year: 2023: 1.
1 paper.