# Consistent Feature Selection for Analytic Deep Neural Networks

arXiv: https://arxiv.org/abs/2010.08097 · HTML: https://mlsanity.com/paper/2010.08097/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 27 of 27)

- [Neural graphical modelling in continuous-time: consistency guarantees and algorithms](https://mlsanity.com/paper/2105.02522/) (2021)
- [A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning](https://mlsanity.com/paper/2311.05877/) (2023)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks](https://mlsanity.com/paper/2110.05667/) (2021)
- [LESS-VFL: Communication-Efficient Feature Selection for Vertical Federated Learning](https://mlsanity.com/paper/2305.02219/) (2023)
- [Provable Identifiability of Two-Layer ReLU Neural Networks via LASSO Regularization](https://mlsanity.com/paper/2305.04267/) (2023)
- [Adaptive Group Lasso Neural Network Models for Functions of Few Variables and Time-Dependent Data](https://mlsanity.com/paper/2108.10825/) (2021)
- [LCEN: A Nonlinear, Interpretable Feature Selection and Machine Learning Algorithm](https://mlsanity.com/paper/2402.17120/) (2024)
- [Interpretable Prediction and Feature Selection for Survival Analysis](https://mlsanity.com/paper/2404.14689/) (2024)
- [Searching for Minimal Optimal Neural Networks](https://mlsanity.com/paper/2109.13061/) (2021)
- [Penalized Generative Variable Selection](https://mlsanity.com/paper/2402.16661/) (2024)
- [Generative adversarial learning with optimal input dimension and its adaptive generator architecture](https://mlsanity.com/paper/2405.03723/) (2024)
- [Comprehend, Divide, and Conquer: Feature Subspace Exploration via Multi-Agent Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2504.17356/) (2025)
- [PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structure](https://mlsanity.com/paper/2511.07997/) (2025)
- [Error Controlled Feature Selection for Ultrahigh Dimensional and Highly Correlated Feature Space Using Deep Learning](https://mlsanity.com/paper/2209.07011/) (2022)
- [Non-Convex Optimizations for Machine Learning with Theoretical Guarantee: Robust Matrix Completion and Neural Network Learning](https://mlsanity.com/paper/2306.16557/) (2023)
- [A Generalization Bound of Deep Neural Networks for Dependent Data](https://mlsanity.com/paper/2310.05892/) (2023)
- [End-to-end Feature Selection Approach for Learning Skinny Trees](https://mlsanity.com/paper/2310.18542/) (2023)
- [A least distance estimator for a multivariate regression model using deep neural networks](https://mlsanity.com/paper/2401.03123/) (2024)
- [Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression](https://mlsanity.com/paper/2406.18137/) (2024)
- [Efficient Model Compression for Bayesian Neural Networks](https://mlsanity.com/paper/2411.00273/) (2024)
- [Simulation-free Structure Learning for Stochastic Population Dynamics](https://mlsanity.com/paper/2510.16656/) (2025)
- [Sparse Bayesian Deep Functional Learning with Structured Region Selection](https://mlsanity.com/paper/2602.20651/) (2026)
- [Knockoffs-based False Discovery Rate Control and Simplification for Deep Neural Networks](https://mlsanity.com/paper/2606.04404/) (2026)
- [Deep Multitask Learning for Mixed-Type Outcomes with Shared Sparsity](https://mlsanity.com/paper/2607.00995/) (2026)
- [Reducing information dependency does not cause training data privacy. Adversarially non-robust features do](https://mlsanity.com/paper/2607.12354/) (2026)
- [Model-Agnostic FDR Control via Group Gaussian Mirror and Permutation SHAP](https://mlsanity.com/paper/2608.00989/) (2026)

## Abstract (arXiv)

One of the most important steps toward interpretability and explainability of neural network models is feature selection, which aims to identify the subset of relevant features. Theoretical results in the field have mostly focused on the prediction aspect of the problem with virtually no work on feature selection consistency for deep neural networks due to the model's severe nonlinearity and unidentifiability. This lack of theoretical foundation casts doubt on the applicability of deep learning to contexts where correct interpretations of the features play a central role. In this work, we investigate the problem of feature selection for analytic deep networks. We prove that for a wide class of networks, including deep feed-forward neural networks, convolutional neural networks, and a major sub-class of residual neural networks, the Adaptive Group Lasso selection procedure with Group Lasso as the base estimator is selection-consistent. The work provides further evidence that Group Lasso might be inefficient for feature selection with neural networks and advocates the use of Adaptive Group Lasso over the popular Group Lasso.