# A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning

arXiv: https://arxiv.org/abs/2311.05877 · HTML: https://mlsanity.com/paper/2311.05877/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 29 of 29)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [LassoNet: A Neural Network with Feature Sparsity](https://mlsanity.com/paper/1907.12207/) (2019)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [The Tree Ensemble Layer: Differentiability meets Conditional Computation](https://mlsanity.com/paper/2002.07772/) (2020)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables](https://mlsanity.com/paper/2303.00918/) (2023)
- [Gradient Boosting Neural Networks: GrowNet](https://mlsanity.com/paper/2002.07971/) (2020)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [DeepPINK: reproducible feature selection in deep neural networks](https://mlsanity.com/paper/1809.01185/) (2018)
- [Consistent Feature Selection for Analytic Deep Neural Networks](https://mlsanity.com/paper/2010.08097/) (2020)
- [Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability](https://mlsanity.com/paper/2108.01335/) (2021)
- [Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon](https://mlsanity.com/paper/1912.01189/) (2019)
- [How good Neural Networks interpretation methods really are? A quantitative benchmark](https://mlsanity.com/paper/2304.02383/) (2023)

## Built on by (most cited first, 24 of 24)

- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks](https://mlsanity.com/paper/2402.11137/) (2024)
- [A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data](https://mlsanity.com/paper/2407.02112/) (2024)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)
- [ERASE: Benchmarking Feature Selection Methods for Deep Recommender Systems](https://mlsanity.com/paper/2403.12660/) (2024)
- [Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework](https://mlsanity.com/paper/2406.08311/) (2024)
- [Indirectly Parameterized Concrete Autoencoders](https://mlsanity.com/paper/2403.00563/) (2024)
- [FeatNavigator: Automatic Feature Augmentation on Tabular Data](https://mlsanity.com/paper/2406.09534/) (2024)
- [Adaptive and Robust Watermark for Generative Tabular Data](https://mlsanity.com/paper/2409.14700/) (2024)
- [Neural Reasoning Networks: Efficient Interpretable Neural Networks With Automatic Textual Explanations](https://mlsanity.com/paper/2410.07966/) (2024)
- [Embedded Federated Feature Selection with Dynamic Sparse Training: Balancing Accuracy-Cost Tradeoffs](https://mlsanity.com/paper/2504.05245/) (2025)
- [Comprehend, Divide, and Conquer: Feature Subspace Exploration via Multi-Agent Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2504.17356/) (2025)
- [How Usable is Automated Feature Engineering for Tabular Data?](https://mlsanity.com/paper/2508.13932/) (2025)
- [Federated Learning With L0 Constraint Via Probabilistic Gates For Sparsity](https://mlsanity.com/paper/2512.23071/) (2025)
- [EntryPrune: Neural Network Feature Selection using First Impressions](https://mlsanity.com/paper/2410.02344/) (2024)
- [Table2Image: Lightweight Tabular Learning with Generated Proxy Representations and Reliability Diagnostics](https://mlsanity.com/paper/2412.06265/) (2024)
- [AutoNFS: Automatic Neural Feature Selection](https://mlsanity.com/paper/2503.13304/) (2025)
- [Automatic and Structure-Aware Sparsification of Hybrid Neural ODEs](https://mlsanity.com/paper/2505.18996/) (2025)
- [Challenges of Heterogeneity in Big Data: A Comparative Study of Classification in Large-Scale Structured and Unstructured Domains](https://mlsanity.com/paper/2512.00298/) (2025)
- [LassoFlexNet: Flexible Neural Architecture for Tabular Data](https://mlsanity.com/paper/2603.20631/) (2026)
- [LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems](https://mlsanity.com/paper/2606.01111/) (2026)
- [Bias in Filter Feature Selection Evaluation: A Meta-Analysis of Datasets, Baselines, and Experimental Design Choices](https://mlsanity.com/paper/2606.07068/) (2026)
- [Adversarial LassoNet: Robust Feature Selection via Stability-Driven Sparse Learning](https://mlsanity.com/paper/2607.03839/) (2026)
- [PACE: Plug-and-Play Contextual Embedding for Feature Screening with Pretrained Tabular Foundation Models](https://mlsanity.com/paper/2609.23574/) (2026)

## Abstract (arXiv)

Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent overfitting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify a reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. Motivated by the increasing popularity of tabular deep learning, we construct a challenging feature selection benchmark evaluated on downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose an input-gradient-based analogue of Lasso for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted or second-order features.