# A Review of Feature Selection Methods Based on Mutual Information

arXiv: https://arxiv.org/abs/1509.07577 · HTML: https://mlsanity.com/paper/1509.07577/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2015.

## Built on by (most cited first, 30 of 67)

- [A Consistent and Efficient Evaluation Strategy for Attribution Methods](https://mlsanity.com/paper/2202.00449/) (2022)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [ZIN: When and How to Learn Invariance Without Environment Partition?](https://mlsanity.com/paper/2203.05818/) (2022)
- [TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks](https://mlsanity.com/paper/2402.11137/) (2024)
- [Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features](https://mlsanity.com/paper/2202.01273/) (2022)
- [Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models](https://mlsanity.com/paper/2509.02859/) (2025)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [FLAC: Fairness-Aware Representation Learning by Suppressing Attribute-Class Associations](https://mlsanity.com/paper/2304.14252/) (2023)
- [Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks](https://mlsanity.com/paper/2311.10609/) (2023)
- [Explainable Deep Neural Network for Multimodal ECG Signals: Intermediate vs Late Fusion](https://mlsanity.com/paper/2508.11666/) (2025)
- [Exact Shapley Attributions in Quadratic-time for FANOVA Gaussian Processes](https://mlsanity.com/paper/2508.14499/) (2025)
- [Evaluating Resilience of Encrypted Traffic Classification Against Adversarial Evasion Attacks](https://mlsanity.com/paper/2105.14564/) (2021)
- [Active Learning and Novel Model Calibration Measurements for Automated Visual Inspection in Manufacturing](https://mlsanity.com/paper/2209.05486/) (2022)
- [A robust estimator of mutual information for deep learning interpretability](https://mlsanity.com/paper/2211.00024/) (2022)
- [FMMI: Flow Matching Mutual Information Estimation](https://mlsanity.com/paper/2511.08552/) (2025)
- [Unsupervised Features Ranking via Coalitional Game Theory for Categorical Data](https://mlsanity.com/paper/2205.09060/) (2022)
- [A survey of some recent developments in measures of association](https://mlsanity.com/paper/2211.04702/) (2022)
- [Causal Entropy and Information Gain for Measuring Causal Control](https://mlsanity.com/paper/2309.07703/) (2023)
- [Enhancing ensemble learning and transfer learning in multimodal data analysis by adaptive dimensionality reduction](https://mlsanity.com/paper/2105.03682/) (2021)
- [Network Generalization Prediction for Safety Critical Tasks in Novel Operating Domains](https://mlsanity.com/paper/2108.07399/) (2021)
- [ManiFeSt: Manifold-based Feature Selection for Small Data Sets](https://mlsanity.com/paper/2207.08574/) (2022)
- [Effective Neural Network L0 Regularization With BinMask](https://mlsanity.com/paper/2304.11237/) (2023)
- [A statistical approach to detect sensitive features in a group fairness setting](https://mlsanity.com/paper/2305.06994/) (2023)
- [Feature Selection: A perspective on inter-attribute cooperation](https://mlsanity.com/paper/2306.16559/) (2023)
- [Binary Feature Mask Optimization for Feature Selection](https://mlsanity.com/paper/2401.12644/) (2024)
- [Active Few-Shot Fine-Tuning](https://mlsanity.com/paper/2402.15441/) (2024)
- [A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions](https://mlsanity.com/paper/2405.02344/) (2024)
- [Data augmentation and feature selection for automatic model recommendation in computational physics](https://mlsanity.com/paper/2101.04530/) (2021)
- [Cost-based feature selection for network model choice](https://mlsanity.com/paper/2101.07766/) (2021)
- [Feature selection for medical diagnosis: Evaluation for using a hybrid Stacked-Genetic approach in the diagnosis of heart disease](https://mlsanity.com/paper/2103.08175/) (2021)

## Abstract (arXiv)

In this work we present a review of the state of the art of information theoretic feature selection methods. The concepts of feature relevance, redundance and complementarity (synergy) are clearly defined, as well as Markov blanket. The problem of optimal feature selection is defined. A unifying theoretical framework is described, which can retrofit successful heuristic criteria, indicating the approximations made by each method. A number of open problems in the field are presented.