# Nonconvex sparse regularization for deep neural networks and its optimality

arXiv: https://arxiv.org/abs/2003.11769 · HTML: https://mlsanity.com/paper/2003.11769/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2020.

## Built on by (most cited first, 21 of 21)

- [A likelihood approach to nonparametric estimation of a singular distribution using deep generative models](https://mlsanity.com/paper/2105.04046/) (2021)
- [Adaptive deep learning for nonlinear time series models](https://mlsanity.com/paper/2207.02546/) (2022)
- [Convergence Rate Analysis of LION](https://mlsanity.com/paper/2411.07724/) (2024)
- [Excess risk bound for deep learning under weak dependence](https://mlsanity.com/paper/2302.07503/) (2023)
- [Sparse-penalized deep neural networks estimator under weak dependence](https://mlsanity.com/paper/2303.01406/) (2023)
- [Non-Asymptotic Guarantees for Robust Statistical Learning under Infinite Variance Assumption](https://mlsanity.com/paper/2201.03182/) (2022)
- [Robust deep learning from weakly dependent data](https://mlsanity.com/paper/2405.05081/) (2024)
- [Intrinsic and extrinsic deep learning on manifolds](https://mlsanity.com/paper/2302.08606/) (2023)
- [Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds](https://mlsanity.com/paper/2410.21702/) (2024)
- [Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality](https://mlsanity.com/paper/2406.08321/) (2024)
- [Function approximation by deep neural networks with parameters 0,± frac12, ± 1, 2](https://mlsanity.com/paper/2103.08659/) (2021)
- [Penalized deep neural networks estimator with general loss functions under weak dependence](https://mlsanity.com/paper/2305.06230/) (2023)
- [A general framework for deep learning](https://mlsanity.com/paper/2512.23425/) (2025)
- [Statistical learning by sparse deep neural networks](https://mlsanity.com/paper/2311.08845/) (2023)
- [Deep Partially Linear Transformation Model for Right-Censored Survival Data](https://mlsanity.com/paper/2412.07611/) (2024)
- [Nonparametric regression with modified ReLU networks](https://mlsanity.com/paper/2207.08306/) (2022)
- [Deep regression learning from dependent observations with minimum error entropy principle](https://mlsanity.com/paper/2603.11138/) (2026)
- [Modeling Covariate Transition for Efficient Estimation of Longitudinal Treatment Effects in Randomized Experiments](https://mlsanity.com/paper/2605.31443/) (2026)
- [Rate-optimal neural boundary detection from unlabeled noisy images](https://mlsanity.com/paper/2606.00715/) (2026)
- [Adaptive deep nonparametric regression from dependent data under covariate shift](https://mlsanity.com/paper/2607.20309/) (2026)
- [Prediction Inference of Time Series with Standard ReLU Deep Neural Networks](https://mlsanity.com/paper/2608.15362/) (2026)

## Abstract (arXiv)

Recent theoretical studies proved that deep neural network (DNN) estimators obtained by minimizing empirical risk with a certain sparsity constraint can attain optimal convergence rates for regression and classification problems. However, the sparsity constraint requires to know certain properties of the true model, which are not available in practice. Moreover, computation is difficult due to the discrete nature of the sparsity constraint. In this paper, we propose a novel penalized estimation method for sparse DNNs, which resolves the aforementioned problems existing in the sparsity constraint. We establish an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derive convergence rates for several learning tasks. In particular, we prove that the sparse-penalized estimator can adaptively attain minimax convergence rates for various nonparametric regression problems. For computation, we develop an efficient gradient-based optimization algorithm that guarantees the monotonic reduction of the objective function.