Unveiling the Power of Sparse Neural Networks for Feature Selection
Not on the map. Why?
Abstract (arXiv)
Sparse Neural Networks (SNNs) have emerged as powerful tools for efficient feature selection. Leveraging the dynamic sparse training (DST) algorithms within SNNs has demonstrated promising feature selection capabilities while drastically reducing computational overheads. Despite these advancements, several critical aspects remain insufficiently explored for feature selection. Questions persist regarding the choice of the DST algorithm for network training, the choice of metric for ranking features/neurons, and the comparative performance of these methods across diverse datasets when compared to dense networks. This paper addresses these gaps by presenting a comprehensive systematic analysis of feature selection with sparse neural networks. Moreover, we introduce a novel metric considering sparse neural network characteristics, which is designed to quantify feature importance within the context of SNNs. Our findings show that feature selection with SNNs trained with DST algorithms can achieve, on average, more than 50% memory and 55% FLOPs reduction compared to the dense networks, while outperforming them in terms of the quality of the selected features. Our code and the supplementary material are available on GitHub (urlhttps://github.com/zahraatashgahi/Neuron-Attribution).
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 47 references that are in the map. First posted on arXiv 8 Aug 2024.
Builds on
28 of 47 references.
References · 28
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2017Improved Training of Wasserstein GANscited by 2,223
- 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
- 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
- 2019AutoML: A Survey of the State-of-the-Artcited by 388
- 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
- 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
- 2020The Hardware Lotterycited by 160
- 2019Concrete Autoencoders for Differentiable Feature Selection and Reconstructioncited by 90
- 2019LassoNet: A Neural Network with Feature Sparsitycited by 81
16 more
- 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
- 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
- 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
- 2018DeepPINK: reproducible feature selection in deep neural networkscited by 41
- 2020Feature Importance Ranking for Deep Learningcited by 31
- 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 27
- 2020FsNet: Feature Selection Network on High-dimensional Biological Datacited by 23
- 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
- 2021Sparse Training Theory for Scalable and Efficient Agentscited by 12
- 2022Composite Feature Selection using Deep Ensemblescited by 11
- 2019A Brain-inspired Algorithm for Training Highly Sparse Neural Networkscited by 8
- 2023Supervised Feature Selection with Neuron Evolution in Sparse Neural Networkscited by 7
- 2022Local Contrastive Feature learning for Tabular Datacited by 4
- 2023Effective Neural Network L0 Regularization With BinMaskcited by 2
- 2023Copula for Instance-wise Feature Selection and Rankingcited by 2
- 2023Embedded feature selection in LSTM networks with multi-objective evolutionary ensemble learning for time series forecastingcited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.