# Unveiling the Power of Sparse Neural Networks for Feature Selection

arXiv: https://arxiv.org/abs/2408.04583 · HTML: https://mlsanity.com/paper/2408.04583/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [LassoNet: A Neural Network with Feature Sparsity](https://mlsanity.com/paper/1907.12207/) (2019)
- [Top-KAST: Top-K Always Sparse Training](https://mlsanity.com/paper/2106.03517/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [DeepPINK: reproducible feature selection in deep neural networks](https://mlsanity.com/paper/1809.01185/) (2018)
- [Feature Importance Ranking for Deep Learning](https://mlsanity.com/paper/2010.08973/) (2020)
- [Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders](https://mlsanity.com/paper/2012.00560/) (2020)
- [FsNet: Feature Selection Network on High-dimensional Biological Data](https://mlsanity.com/paper/2001.08322/) (2020)
- [Where to Pay Attention in Sparse Training for Feature Selection?](https://mlsanity.com/paper/2211.14627/) (2022)
- [Sparse Training Theory for Scalable and Efficient Agents](https://mlsanity.com/paper/2103.01636/) (2021)
- [Composite Feature Selection using Deep Ensembles](https://mlsanity.com/paper/2211.00631/) (2022)
- [A Brain-inspired Algorithm for Training Highly Sparse Neural Networks](https://mlsanity.com/paper/1903.07138/) (2019)
- [Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks](https://mlsanity.com/paper/2303.07200/) (2023)
- [Local Contrastive Feature learning for Tabular Data](https://mlsanity.com/paper/2211.10549/) (2022)
- [Effective Neural Network L0 Regularization With BinMask](https://mlsanity.com/paper/2304.11237/) (2023)
- [Copula for Instance-wise Feature Selection and Ranking](https://mlsanity.com/paper/2308.00549/) (2023)
- [Embedded feature selection in LSTM networks with multi-objective evolutionary ensemble learning for time series forecasting](https://mlsanity.com/paper/2312.17517/) (2023)

## Built on by (most cited first, 3 of 3)

- [Exploring Neural Granger Causality with xLSTMs: Unveiling Temporal Dependencies in Complex Data](https://mlsanity.com/paper/2502.09981/) (2025)
- [Sparse Self-Federated Learning for Energy Efficient Cooperative Intelligence in Society 5.0](https://mlsanity.com/paper/2507.07613/) (2025)
- [EntryPrune: Neural Network Feature Selection using First Impressions](https://mlsanity.com/paper/2410.02344/) (2024)

## Abstract (arXiv)

Sparse Neural Networks (SNNs) have emerged as powerful tools for efficient feature selection. Leveraging the dynamic sparse training (DST) algorithms within SNNs has demonstrated promising feature selection capabilities while drastically reducing computational overheads. Despite these advancements, several critical aspects remain insufficiently explored for feature selection. Questions persist regarding the choice of the DST algorithm for network training, the choice of metric for ranking features/neurons, and the comparative performance of these methods across diverse datasets when compared to dense networks. This paper addresses these gaps by presenting a comprehensive systematic analysis of feature selection with sparse neural networks. Moreover, we introduce a novel metric considering sparse neural network characteristics, which is designed to quantify feature importance within the context of SNNs. Our findings show that feature selection with SNNs trained with DST algorithms can achieve, on average, more than 50% memory and 55% FLOPs reduction compared to the dense networks, while outperforming them in terms of the quality of the selected features. Our code and the supplementary material are available on GitHub (urlhttps://github.com/zahraatashgahi/Neuron-Attribution).