# Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

arXiv: https://arxiv.org/abs/2406.18137 · HTML: https://mlsanity.com/paper/2406.18137/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [Pathological spectra of the Fisher information metric and its variants in deep neural networks](https://mlsanity.com/paper/1910.05992/) (2019)
- [Consistent Feature Selection for Analytic Deep Neural Networks](https://mlsanity.com/paper/2010.08097/) (2020)
- [Statistical guarantees for sparse deep learning](https://mlsanity.com/paper/2212.05427/) (2022)
- [Consistent feature selection for neural networks via Adaptive Group Lasso](https://mlsanity.com/paper/2006.00334/) (2020)
- [Provable Identifiability of Two-Layer ReLU Neural Networks via LASSO Regularization](https://mlsanity.com/paper/2305.04267/) (2023)
- [Sparse-Input Neural Network using Group Concave Regularization](https://mlsanity.com/paper/2307.00344/) (2023)
- [Double Sparsity Kernel Learning with Automatic Variable Selection and Data Extraction](https://mlsanity.com/paper/1706.01426/) (2017)
- [Statistical learning by sparse deep neural networks](https://mlsanity.com/paper/2311.08845/) (2023)

## Abstract (arXiv)

Generalization theory has been established for sparse deep neural networks under high-dimensional regime. Beyond generalization, parameter estimation is also important since it is crucial for variable selection and interpretability of deep neural networks. Current theoretical studies concerning parameter estimation mainly focus on two-layer neural networks, which is due to the fact that the convergence of parameter estimation heavily relies on the regularity of the Hessian matrix, while the Hessian matrix of deep neural networks is highly singular. To avoid the unidentifiability of deep neural networks in parameter estimation, we propose to conduct nonparametric estimation of partial derivatives with respect to inputs. We first show that model convergence of sparse deep neural networks is guaranteed in that the sample complexity only grows with the logarithm of the number of parameters or the input dimension when the ℓ1-norm of parameters is well constrained. Then by bounding the norm and the divergence of partial derivatives, we establish that the convergence rate of nonparametric estimation of partial derivatives scales as O(n-1/4), a rate which is slower than the model convergence rate O(n-1/2). To the best of our knowledge, this study combines nonparametric estimation and parametric sparse deep neural networks for the first time. As nonparametric estimation of partial derivatives is of great significance for nonlinear variable selection, the current results show the promising future for the interpretability of deep neural networks.