# Forward-Backward Selection with Early Dropping

arXiv: https://arxiv.org/abs/1705.10770 · HTML: https://mlsanity.com/paper/1705.10770/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2017.

## Built on by (most cited first, 18 of 18)

- [Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning](https://mlsanity.com/paper/2201.03529/) (2022)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis](https://mlsanity.com/paper/2305.00090/) (2023)
- [Sparse Bayesian Learning via Stepwise Regression](https://mlsanity.com/paper/2106.06095/) (2021)
- [A model-free feature selection technique of feature screening and random forest based recursive feature elimination](https://mlsanity.com/paper/2302.07449/) (2023)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [On the Generalizability of Motion Models for Road Users in Heterogeneous Shared Traffic Spaces](https://mlsanity.com/paper/2101.06974/) (2021)
- [Fair Causal Feature Selection](https://mlsanity.com/paper/2306.10336/) (2023)
- [Quantum Algorithms for the Pathwise Lasso](https://mlsanity.com/paper/2312.14141/) (2023)
- [Feature Selection from Differentially Private Correlations](https://mlsanity.com/paper/2408.10862/) (2024)
- [Inherently Interpretable Tree Ensemble Learning](https://mlsanity.com/paper/2410.19098/) (2024)
- [Parallel feature selection based on the trace ratio criterion](https://mlsanity.com/paper/2203.01635/) (2022)
- [LaPLACE: Probabilistic Local Model-Agnostic Causal Explanations](https://mlsanity.com/paper/2310.00570/) (2023)
- [Towards Automated Causal Discovery: a case study on 5G telecommunication data](https://mlsanity.com/paper/2402.14481/) (2024)
- [Coordinated Multi-Neighborhood Learning on a Directed Acyclic Graph](https://mlsanity.com/paper/2405.15358/) (2024)
- [A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders](https://mlsanity.com/paper/2606.07007/) (2026)
- [Quantum Algorithm for Sparse Online Learning with Truncated Gradient Descent](https://mlsanity.com/paper/2411.03925/) (2024)
- [Towards Interpretable and Efficient Feature Selection in Trajectory Datasets: A Taxonomic Approach](https://mlsanity.com/paper/2506.20359/) (2025)

## Abstract (arXiv)

Forward-backward selection is one of the most basic and commonly-used feature selection algorithms available. It is also general and conceptually applicable to many different types of data. In this paper, we propose a heuristic that significantly improves its running time, while preserving predictive accuracy. The idea is to temporarily discard the variables that are conditionally independent with the outcome given the selected variable set. Depending on how those variables are reconsidered and reintroduced, this heuristic gives rise to a family of algorithms with increasingly stronger theoretical guarantees. In distributions that can be faithfully represented by Bayesian networks or maximal ancestral graphs, members of this algorithmic family are able to correctly identify the Markov blanket in the sample limit. In experiments we show that the proposed heuristic increases computational efficiency by about two orders of magnitude in high-dimensional problems, while selecting fewer variables and retaining predictive performance. Furthermore, we show that the proposed algorithm and feature selection with LASSO perform similarly when restricted to select the same number of variables, making the proposed algorithm an attractive alternative for problems where no (efficient) algorithm for LASSO exists.