# Feature Selection from High-Dimensional Data with Very Low Sample Size: A Cautionary Tale

arXiv: https://arxiv.org/abs/2008.12025 · HTML: https://mlsanity.com/paper/2008.12025/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2020.

## Built on by (most cited first, 3 of 3)

- [Random Forest Kernel for High-Dimension Low Sample Size Classification](https://mlsanity.com/paper/2310.14710/) (2023)
- [Experimental Design for Multi-Channel Imaging via Task-Driven Feature Selection](https://mlsanity.com/paper/2210.06891/) (2022)
- [Evolutionary Optimization for the Classification of Small Molecules Regulating the Circadian Rhythm Period: A Reliable Assessment](https://mlsanity.com/paper/2505.05485/) (2025)

## Abstract (arXiv)

In classification problems, the purpose of feature selection is to identify a small, highly discriminative subset of the original feature set. In many applications, the dataset may have thousands of features and only a few dozens of samples (sometimes termed `wide'). This study is a cautionary tale demonstrating why feature selection in such cases may lead to undesirable results. In view to highlight the sample size issue, we derive the required sample size for declaring two features different. Using an example, we illustrate the heavy dependency between feature set and classifier, which poses a question to classifier-agnostic feature selection methods. However, the choice of a good selector-classifier pair is hampered by the low correlation between estimated and true error rate, as illustrated by another example. While previous studies raising similar issues validate their message with mostly synthetic data, here we carried out an experiment with 20 real datasets. We created an exaggerated scenario whereby we cut a very small portion of the data (10 instances per class) for feature selection and used the rest of the data for testing. The results reinforce the caution and suggest that it may be better to refrain from feature selection from very wide datasets rather than return misleading output to the user.