# Top-k Regularization for Supervised Feature Selection

arXiv: https://arxiv.org/abs/2106.02197 · HTML: https://mlsanity.com/paper/2106.02197/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 44 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [Feature Importance Ranking for Deep Learning](https://mlsanity.com/paper/2010.08973/) (2020)
- [FsNet: Feature Selection Network on High-dimensional Biological Data](https://mlsanity.com/paper/2001.08322/) (2020)
- [Infinite Feature Selection: A Graph-based Feature Filtering Approach](https://mlsanity.com/paper/2006.08184/) (2020)
- [Ultra High-Dimensional Nonlinear Feature Selection for Big Biological Data](https://mlsanity.com/paper/1608.04048/) (2016)

## Abstract (arXiv)

Feature selection identifies subsets of informative features and reduces dimensions in the original feature space, helping provide insights into data generation or a variety of domain problems. Existing methods mainly depend on feature scoring functions or sparse regularizations; nonetheless, they have limited ability to reconcile the representativeness and inter-correlations of features. In this paper, we introduce a novel, simple yet effective regularization approach, named top-k regularization, to supervised feature selection in regression and classification tasks. Structurally, the top-k regularization induces a sub-architecture on the architecture of a learning model to boost its ability to select the most informative features and model complex nonlinear relationships simultaneously. Theoretically, we derive and mathematically prove a uniform approximation error bound for using this approach to approximate high-dimensional sparse functions. Extensive experiments on a wide variety of benchmarking datasets show that the top-k regularization is effective and stable for supervised feature selection.