# Provable Identifiability of Two-Layer ReLU Neural Networks via LASSO Regularization

arXiv: https://arxiv.org/abs/2305.04267 · HTML: https://mlsanity.com/paper/2305.04267/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 7 May 2023.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Learning One-hidden-layer Neural Networks with Landscape Design](https://mlsanity.com/paper/1711.00501/) (2017)
- [Model Selection Techniques -- An Overview](https://mlsanity.com/paper/1810.09583/) (2018)
- [Consistent Feature Selection for Analytic Deep Neural Networks](https://mlsanity.com/paper/2010.08097/) (2020)
- [Pruning Deep Neural Networks from a Sparsity Perspective](https://mlsanity.com/paper/2302.05601/) (2023)
- [Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation](https://mlsanity.com/paper/1902.00800/) (2019)
- [On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition](https://mlsanity.com/paper/1802.07301/) (2018)
- [A Theoretical Understanding of Neural Network Compression from Sparse Linear Approximation](https://mlsanity.com/paper/2206.05604/) (2022)
- [The Rate of Convergence of Variation-Constrained Deep Neural Networks](https://mlsanity.com/paper/2106.12068/) (2021)

## Built on by (most cited first, 4 of 4)

- [Unraveling the Gradient Descent Dynamics of Transformers](https://mlsanity.com/paper/2411.07538/) (2024)
- [Demystifying Poisoning Backdoor Attacks from a Statistical Perspective](https://mlsanity.com/paper/2310.10780/) (2023)
- [Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression](https://mlsanity.com/paper/2406.18137/) (2024)
- [Towards Atoms of Large Language Models](https://mlsanity.com/paper/2509.20784/) (2025)

## Abstract (arXiv)

LASSO regularization is a popular regression tool to enhance the prediction accuracy of statistical models by performing variable selection through the ℓ1 penalty, initially formulated for the linear model and its variants. In this paper, the territory of LASSO is extended to two-layer ReLU neural networks, a fashionable and powerful nonlinear regression model. Specifically, given a neural network whose output y depends only on a small subset of input boldsymbolx, denoted by Sstar, we prove that the LASSO estimator can stably reconstruct the neural network and identify Sstar when the number of samples scales logarithmically with the input dimension. This challenging regime has been well understood for linear models while barely studied for neural networks. Our theory lies in an extended Restricted Isometry Property (RIP)-based analysis framework for two-layer ReLU neural networks, which may be of independent interest to other LASSO or neural network settings. Based on the result, we advocate a neural network-based variable selection method. Experiments on simulated and real-world datasets show promising performance of the variable selection approach compared with existing techniques.