# Nonconvex regularization for sparse neural networks

arXiv: https://arxiv.org/abs/2004.11515 · HTML: https://mlsanity.com/paper/2004.11515/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 7 of 7)

- [Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisation](https://mlsanity.com/paper/2402.01052/) (2024)
- [Optimal Regularization for a Data Source](https://mlsanity.com/paper/2212.13597/) (2022)
- [Provably Convergent Data-Driven Convex-Nonconvex Regularization](https://mlsanity.com/paper/2310.05812/) (2023)
- [Solving Nonlinear PDEs with Sparse Radial Basis Function Networks](https://mlsanity.com/paper/2505.07765/) (2025)
- [Nonuniform random feature models using derivative information](https://mlsanity.com/paper/2410.02132/) (2024)
- [Sparse RBF Networks for PDEs and nonlocal equations: function space theory, operator calculus, and training algorithms](https://mlsanity.com/paper/2601.17562/) (2026)
- [Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layers](https://mlsanity.com/paper/2609.21126/) (2026)

## Abstract (arXiv)

Convex ℓ1 regularization using an infinite dictionary of neurons has been suggested for constructing neural networks with desired approximation guarantees, but can be affected by an arbitrary amount of over-parametrization. This can lead to a loss of sparsity and result in networks with too many active neurons for the given data, in particular if the number of data samples is large. As a remedy, in this paper, a nonconvex regularization method is investigated in the context of shallow ReLU networks: We prove that in contrast to the convex approach, any resulting (locally optimal) network is finite even in the presence of infinite data (i.e., if the data distribution is known and the limiting case of infinite samples is considered). Moreover, we show that approximation guarantees and existing bounds on the network size for finite data are maintained.