# Feature Selection via Mutual Information: New Theoretical Insights

arXiv: https://arxiv.org/abs/1907.07384 · HTML: https://mlsanity.com/paper/1907.07384/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2019.

## Built on by (most cited first, 17 of 17)

- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training](https://mlsanity.com/paper/2402.09050/) (2024)
- [Few-shot Action Recognition via Intra- and Inter-Video Information Maximization](https://mlsanity.com/paper/2305.06114/) (2023)
- [ARLO: A Framework for Automated Reinforcement Learning](https://mlsanity.com/paper/2205.10416/) (2022)
- [Causal Entropy and Information Gain for Measuring Causal Control](https://mlsanity.com/paper/2309.07703/) (2023)
- [Causal Feature Selection via Transfer Entropy](https://mlsanity.com/paper/2310.11059/) (2023)
- [MoJE: Mixture of Jailbreak Experts, Naive Tabular Classifiers as Guard for Prompt Attacks](https://mlsanity.com/paper/2409.17699/) (2024)
- [Active Few-Shot Fine-Tuning](https://mlsanity.com/paper/2402.15441/) (2024)
- [Capturing the security expert knowledge in feature selection for web application attack detection](https://mlsanity.com/paper/2407.18445/) (2024)
- [A Notion of Feature Importance by Decorrelation and Detection of Trends by Random Forest Regression](https://mlsanity.com/paper/2303.01156/) (2023)
- [Compressive Feature Selection for Remote Visual Multi-Task Inference](https://mlsanity.com/paper/2405.09077/) (2024)
- [Movement Prediction-Adjusted Naive Forecast: Is the Naive Baseline Unbeatable in Financial Time Series Forecasting?](https://mlsanity.com/paper/2406.14469/) (2024)
- [Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information](https://mlsanity.com/paper/2502.11260/) (2025)
- [Predicting Fetal Birthweight from High Dimensional Data using Advanced Machine Learning](https://mlsanity.com/paper/2502.14270/) (2025)
- [A Scalable Hierarchical Intrusion Detection System for Internet of Vehicles](https://mlsanity.com/paper/2505.16215/) (2025)
- [nCMD: Benign-Anchored Feature Selection for Imbalanced Network Intrusion Detection](https://mlsanity.com/paper/2606.09934/) (2026)
- [BARS: Benign-Anchored Ranking and Selection for False Alarm Reduction in Network Intrusion Detection](https://mlsanity.com/paper/2607.13203/) (2026)

## Abstract (arXiv)

Mutual information has been successfully adopted in filter feature-selection methods to assess both the relevancy of a subset of features in predicting the target variable and the redundancy with respect to other variables. However, existing algorithms are mostly heuristic and do not offer any guarantee on the proposed solution. In this paper, we provide novel theoretical results showing that conditional mutual information naturally arises when bounding the ideal regression/classification errors achieved by different subsets of features. Leveraging on these insights, we propose a novel stopping condition for backward and forward greedy methods which ensures that the ideal prediction error using the selected feature subset remains bounded by a user-specified threshold. We provide numerical simulations to support our theoretical claims and compare to common heuristic methods.