# Learning Compact Neural Networks with Regularization

arXiv: https://arxiv.org/abs/1802.01223 · HTML: https://mlsanity.com/paper/1802.01223/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2018.

## Built on by (most cited first, 8 of 8)

- [Generalization Guarantees for Neural Architecture Search with Train-Validation Split](https://mlsanity.com/paper/2104.14132/) (2021)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [Outlier Robust and Sparse Estimation of Linear Regression Coefficients](https://mlsanity.com/paper/2208.11592/) (2022)
- [TEDDY: Trimming Edges with Degree-based Discrimination strategY](https://mlsanity.com/paper/2402.01261/) (2024)
- [Provable Pathways: Learning Multiple Tasks over Multiple Paths](https://mlsanity.com/paper/2303.04338/) (2023)
- [NODE-SELECT: A Graph Neural Network Based On A Selective Propagation Technique](https://mlsanity.com/paper/2102.08588/) (2021)
- [An Iteratively Reweighted Method for Sparse Optimization on Nonconvex ℓp Ball](https://mlsanity.com/paper/2104.02912/) (2021)
- [Estimation of sparse linear regression coefficients under L-subexponential covariates](https://mlsanity.com/paper/2304.11958/) (2023)

## Abstract (arXiv)

Proper regularization is critical for speeding up training, improving generalization performance, and learning compact models that are cost efficient. We propose and analyze regularized gradient descent algorithms for learning shallow neural networks. Our framework is general and covers weight-sharing (convolutional networks), sparsity (network pruning), and low-rank constraints among others. We first introduce covering dimension to quantify the complexity of the constraint set and provide insights on the generalization properties. Then, we show that proposed algorithms become well-behaved and local linear convergence occurs once the amount of data exceeds the covering dimension. Overall, our results demonstrate that near-optimal sample complexity is sufficient for efficient learning and illustrate how regularization can be beneficial to learn over-parameterized networks.