# Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification

arXiv: https://arxiv.org/abs/1711.07592 · HTML: https://mlsanity.com/paper/1711.07592/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2017.

## Built on by (most cited first, 30 of 39)

- [Learning to Maximize Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2301.00557/) (2023)
- [Locally Sparse Neural Networks for Tabular Biomedical Data](https://mlsanity.com/paper/2106.06468/) (2021)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [Exploring Large Language Models for Feature Selection: A Data-centric Perspective](https://mlsanity.com/paper/2408.12025/) (2024)
- [Explaining Time Series via Contrastive and Locally Sparse Perturbations](https://mlsanity.com/paper/2401.08552/) (2024)
- [Estimating Conditional Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2306.03301/) (2023)
- [How to Find a Good Explanation for Clustering?](https://mlsanity.com/paper/2112.06580/) (2021)
- [Statistical guarantees for sparse deep learning](https://mlsanity.com/paper/2212.05427/) (2022)
- [Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration](https://mlsanity.com/paper/2110.00653/) (2021)
- [Conditional Distribution Function Estimation Using Neural Networks for Censored and Uncensored Data](https://mlsanity.com/paper/2207.02384/) (2022)
- [GCondNet: A Novel Method for Improving Neural Networks on Small High-Dimensional Tabular Data](https://mlsanity.com/paper/2211.06302/) (2022)
- [Weight Predictor Network with Feature Selection for Small Sample Tabular Biomedical Data](https://mlsanity.com/paper/2211.15616/) (2022)
- [The Contextual Lasso: Sparse Linear Models via Deep Neural Networks](https://mlsanity.com/paper/2302.00878/) (2023)
- [Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks](https://mlsanity.com/paper/2106.02260/) (2021)
- [An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity](https://mlsanity.com/paper/2205.05428/) (2022)
- [Representation Retrieval Learning for Heterogeneous Data Integration](https://mlsanity.com/paper/2503.09494/) (2025)
- [Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guarantees](https://mlsanity.com/paper/2204.07293/) (2022)
- [DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation](https://mlsanity.com/paper/2211.05499/) (2022)
- [Sparse-Input Neural Network using Group Concave Regularization](https://mlsanity.com/paper/2307.00344/) (2023)
- [Learning Channel Importance for High Content Imaging with Interpretable Deep Input Channel Mixing](https://mlsanity.com/paper/2308.16637/) (2023)
- [Feature Selection in the Contrastive Analysis Setting](https://mlsanity.com/paper/2310.18531/) (2023)
- [A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks](https://mlsanity.com/paper/2201.08652/) (2022)
- [Regularization can make diffusion models more efficient](https://mlsanity.com/paper/2502.09151/) (2025)
- [Lockout: Sparse Regularization of Neural Networks](https://mlsanity.com/paper/2107.07160/) (2021)
- [Searching for Minimal Optimal Neural Networks](https://mlsanity.com/paper/2109.13061/) (2021)
- [SLM: End-to-end Feature Selection via Sparse Learnable Masks](https://mlsanity.com/paper/2304.03202/) (2023)
- [DeRisk: An Effective Deep Learning Framework for Credit Risk Prediction over Real-World Financial Data](https://mlsanity.com/paper/2308.03704/) (2023)
- [Penalized Generative Variable Selection](https://mlsanity.com/paper/2402.16661/) (2024)
- [PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structure](https://mlsanity.com/paper/2511.07997/) (2025)
- [LLM-Driven Reasoning for Constraint-Aware Feature Selection in Industrial Systems](https://mlsanity.com/paper/2603.24979/) (2026)

## Abstract (arXiv)

Neural networks are usually not the tool of choice for nonparametric high-dimensional problems where the number of input features is much larger than the number of observations. Though neural networks can approximate complex multivariate functions, they generally require a large number of training observations to obtain reasonable fits, unless one can learn the appropriate network structure. In this manuscript, we show that neural networks can be applied successfully to high-dimensional settings if the true function falls in a low dimensional subspace, and proper regularization is used. We propose fitting a neural network with a sparse group lasso penalty on the first-layer input weights. This results in a neural net that only uses a small subset of the original features. In addition, we characterize the statistical convergence of the penalized empirical risk minimizer to the optimal neural network: we show that the excess risk of this penalized estimator only grows with the logarithm of the number of input features; and we show that the weights of irrelevant features converge to zero. Via simulation studies and data analyses, we show that these sparse-input neural networks outperform existing nonparametric high-dimensional estimation methods when the data has complex higher-order interactions.