# Variational Information Maximization for Feature Selection

arXiv: https://arxiv.org/abs/1606.02827 · HTML: https://mlsanity.com/paper/1606.02827/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2016.

## Built on by (most cited first, 7 of 7)

- [Mutual Information Preserving Neural Network Pruning](https://mlsanity.com/paper/2411.00147/) (2024)
- [Partial Information Decomposition for Data Interpretability and Feature Selection](https://mlsanity.com/paper/2405.19212/) (2024)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [Feature Selection for Network Intrusion Detection](https://mlsanity.com/paper/2411.11603/) (2024)
- [Improving Mutual Information based Feature Selection by Boosting Unique Relevance](https://mlsanity.com/paper/2212.06143/) (2022)
- [Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks](https://mlsanity.com/paper/2106.10252/) (2021)
- [Optimizing Feature Extraction for On-device Model Inference with User Behavior Sequences](https://mlsanity.com/paper/2603.21508/) (2026)

## Abstract (arXiv)

Feature selection is one of the most fundamental problems in machine learning. An extensive body of work on information-theoretic feature selection exists which is based on maximizing mutual information between subsets of features and class labels. Practical methods are forced to rely on approximations due to the difficulty of estimating mutual information. We demonstrate that approximations made by existing methods are based on unrealistic assumptions. We formulate a more flexible and general class of assumptions based on variational distributions and use them to tractably generate lower bounds for mutual information. These bounds define a novel information-theoretic framework for feature selection, which we prove to be optimal under tree graphical models with proper choice of variational distributions. Our experiments demonstrate that the proposed method strongly outperforms existing information-theoretic feature selection approaches.