# Where to Pay Attention in Sparse Training for Feature Selection?

arXiv: https://arxiv.org/abs/2211.14627 · HTML: https://mlsanity.com/paper/2211.14627/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 82 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [Top-KAST: Top-K Always Sparse Training](https://mlsanity.com/paper/2106.03517/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better](https://mlsanity.com/paper/2112.09824/) (2021)
- [SpaceNet: Make Free Space For Continual Learning](https://mlsanity.com/paper/2007.07617/) (2020)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware](https://mlsanity.com/paper/1901.09181/) (2019)
- [Sparse Weight Activation Training](https://mlsanity.com/paper/2001.01969/) (2020)
- [Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity](https://mlsanity.com/paper/2106.14568/) (2021)
- [Multi-objective Evolutionary Federated Learning](https://mlsanity.com/paper/1812.07478/) (2018)
- [Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders](https://mlsanity.com/paper/2012.00560/) (2020)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [Survey on Feature Selection](https://mlsanity.com/paper/1510.02892/) (2015)
- [FsNet: Feature Selection Network on High-dimensional Biological Data](https://mlsanity.com/paper/2001.08322/) (2020)
- [Topological Insights into Sparse Neural Networks](https://mlsanity.com/paper/2006.14085/) (2020)
- [Kernel Feature Selection via Conditional Covariance Minimization](https://mlsanity.com/paper/1707.01164/) (2017)
- [Truly Sparse Neural Networks at Scale](https://mlsanity.com/paper/2102.01732/) (2021)
- [Sparse Training Theory for Scalable and Efficient Agents](https://mlsanity.com/paper/2103.01636/) (2021)
- [Learning to Recognize Musical Genre from Audio](https://mlsanity.com/paper/1803.05337/) (2018)

## Built on by (most cited first, 15 of 15)

- [Interpretable Deep Clustering for Tabular Data](https://mlsanity.com/paper/2306.04785/) (2023)
- [E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2312.04727/) (2023)
- [Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.06548/) (2023)
- [Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness](https://mlsanity.com/paper/2410.03030/) (2024)
- [Bridging Trustworthiness and Open-World Learning: An Exploratory Neural Approach for Enhancing Interpretability, Generalization, and Robustness](https://mlsanity.com/paper/2308.03666/) (2023)
- [Indirectly Parameterized Concrete Autoencoders](https://mlsanity.com/paper/2403.00563/) (2024)
- [Unveiling the Power of Sparse Neural Networks for Feature Selection](https://mlsanity.com/paper/2408.04583/) (2024)
- [Copula for Instance-wise Feature Selection and Ranking](https://mlsanity.com/paper/2308.00549/) (2023)
- [Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution](https://mlsanity.com/paper/2505.24037/) (2025)
- [Joint Feature and Differentiable  k -NN Graph Learning using Dirichlet Energy](https://mlsanity.com/paper/2305.12396/) (2023)
- [Embedded Federated Feature Selection with Dynamic Sparse Training: Balancing Accuracy-Cost Tradeoffs](https://mlsanity.com/paper/2504.05245/) (2025)
- [SAND: One-Shot Feature Selection with Additive Noise Distortion](https://mlsanity.com/paper/2505.03923/) (2025)
- [EntryPrune: Neural Network Feature Selection using First Impressions](https://mlsanity.com/paper/2410.02344/) (2024)
- [AdaS&S: a One-Shot Supernet Approach for Automatic Embedding Size Search in Deep Recommender System](https://mlsanity.com/paper/2411.07504/) (2024)
- [You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2504.12471/) (2025)

## Abstract (arXiv)

A new line of research for feature selection based on neural networks has recently emerged. Despite its superiority to classical methods, it requires many training iterations to converge and detect informative features. The computational time becomes prohibitively long for datasets with a large number of samples or a very high dimensional feature space. In this paper, we present a new efficient unsupervised method for feature selection based on sparse autoencoders. In particular, we propose a new sparse training algorithm that optimizes a model's sparse topology during training to pay attention to informative features quickly. The attention-based adaptation of the sparse topology enables fast detection of informative features after a few training iterations. We performed extensive experiments on 10 datasets of different types, including image, speech, text, artificial, and biological. They cover a wide range of characteristics, such as low and high-dimensional feature spaces, and few and large training samples. Our proposed approach outperforms the state-of-the-art methods in terms of selecting informative features while reducing training iterations and computational costs substantially. Moreover, the experiments show the robustness of our method in extremely noisy environments.