# Searching for Minimal Optimal Neural Networks

arXiv: https://arxiv.org/abs/2109.13061 · HTML: https://mlsanity.com/paper/2109.13061/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [CondenseNet: An Efficient DenseNet using Learned Group Convolutions](https://mlsanity.com/paper/1711.09224/) (2017)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [Deep Neural Networks for Estimation and Inference](https://mlsanity.com/paper/1809.09953/) (2018)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [Consistent Feature Selection for Analytic Deep Neural Networks](https://mlsanity.com/paper/2010.08097/) (2020)
- [Auto-Sizing Neural Networks: With Applications to n-gram Language Models](https://mlsanity.com/paper/1508.05051/) (2015)
- [Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation](https://mlsanity.com/paper/1910.06717/) (2019)
- [Consistent feature selection for neural networks via Adaptive Group Lasso](https://mlsanity.com/paper/2006.00334/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Generalization Bound of Deep Neural Networks for Dependent Data](https://mlsanity.com/paper/2310.05892/) (2023)

## Abstract (arXiv)

Large neural network models have high predictive power but may suffer from overfitting if the training set is not large enough. Therefore, it is desirable to select an appropriate size for neural networks. The destructive approach, which starts with a large architecture and then reduces the size using a Lasso-type penalty, has been used extensively for this task. Despite its popularity, there is no theoretical guarantee for this technique. Based on the notion of minimal neural networks, we posit a rigorous mathematical framework for studying the asymptotic theory of the destructive technique. We prove that Adaptive group Lasso is consistent and can reconstruct the correct number of hidden nodes of one-hidden-layer feedforward networks with high probability. To the best of our knowledge, this is the first theoretical result establishing for the destructive technique.