# Learning to Maximize Mutual Information for Dynamic Feature Selection

arXiv: https://arxiv.org/abs/2301.00557 · HTML: https://mlsanity.com/paper/2301.00557/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 72 references that are in the map.
- First posted on arXiv 2 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Learning to Explain: An Information-Theoretic Perspective on Model Interpretation](https://mlsanity.com/paper/1802.07814/) (2018)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE](https://mlsanity.com/paper/1809.11142/) (2018)
- [VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data](https://mlsanity.com/paper/2006.11941/) (2020)
- [Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations](https://mlsanity.com/paper/2103.01890/) (2021)
- [Active Feature Acquisition with Generative Surrogate Models](https://mlsanity.com/paper/2010.02433/) (2020)
- [Restricted Strong Convexity Implies Weak Submodularity](https://mlsanity.com/paper/1612.00804/) (2016)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [Classification with Costly Features using Deep Reinforcement Learning](https://mlsanity.com/paper/1711.07364/) (2017)
- [Interpretable by Design: Learning Predictors by Composing Interpretable Queries](https://mlsanity.com/paper/2207.00938/) (2022)
- [Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data Streams](https://mlsanity.com/paper/1901.00243/) (2019)
- [Dropout Feature Ranking for Deep Learning Models](https://mlsanity.com/paper/1712.08645/) (2017)
- [Active Information Acquisition](https://mlsanity.com/paper/1602.02181/) (2016)
- [Dynamic Question Ordering in Online Surveys](https://mlsanity.com/paper/1607.04209/) (2016)

## Built on by (most cited first, 30 of 56)

- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [Estimating Conditional Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2306.03301/) (2023)
- [Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution](https://mlsanity.com/paper/2401.15866/) (2024)
- [A Survey on Active Feature Acquisition Strategies](https://mlsanity.com/paper/2502.11067/) (2025)
- [Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection](https://mlsanity.com/paper/2602.08003/) (2026)
- [Stochastic Encodings for Active Feature Acquisition](https://mlsanity.com/paper/2508.01957/) (2025)
- [AFABench: A Generic Framework for Benchmarking Active Feature Acquisition](https://mlsanity.com/paper/2508.14734/) (2025)
- [All models are wrong, some are useful: Model Selection with Limited Labels](https://mlsanity.com/paper/2410.13609/) (2024)
- [Contextual Feature Selection with Conditional Stochastic Gates](https://mlsanity.com/paper/2312.14254/) (2023)
- [Conformal Information Pursuit for Interactively Guiding Large Language Models](https://mlsanity.com/paper/2507.03279/) (2025)
- [Information-Theoretic Distillation for Reference-less Summarization](https://mlsanity.com/paper/2403.13780/) (2024)
- [Learning-To-Measure: In-Context Active Feature Acquisition](https://mlsanity.com/paper/2510.12624/) (2025)
- [Partial Information Decomposition for Data Interpretability and Feature Selection](https://mlsanity.com/paper/2405.19212/) (2024)
- [Feature Selection in the Contrastive Analysis Setting](https://mlsanity.com/paper/2310.18531/) (2023)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [Constrained Multiview Representation for Self-supervised Contrastive Learning](https://mlsanity.com/paper/2402.03456/) (2024)
- [Adaptive and Robust Watermark for Generative Tabular Data](https://mlsanity.com/paper/2409.14700/) (2024)
- [Feature Selection for Network Intrusion Detection](https://mlsanity.com/paper/2411.11603/) (2024)
- [MINT: A wrapper to make multi-modal and multi-image AI models interactive](https://mlsanity.com/paper/2401.12032/) (2024)
- [Predicting Through Generation: Why Generation Is Better for Prediction](https://mlsanity.com/paper/2502.17817/) (2025)
- [IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports](https://mlsanity.com/paper/2505.00191/) (2025)
- [From Pixels to Perception: Interpretable Predictions via Instance-wise Grouped Feature Selection](https://mlsanity.com/paper/2505.06003/) (2025)
- [PRECISE-AS: Personalized Reinforcement Learning for Efficient Point-of-Care Echocardiography in Aortic Stenosis Diagnosis](https://mlsanity.com/paper/2509.02898/) (2025)
- [A distributed neural network architecture for dynamic sensor selection with application to bandwidth-constrained body-sensor networks](https://mlsanity.com/paper/2308.08379/) (2023)
- [Local Feature Selection without Label or Feature Leakage for Interpretable Machine Learning Predictions](https://mlsanity.com/paper/2407.11778/) (2024)
- [Dynamic Feature Selection from Variable Feature Sets Using Features of Features](https://mlsanity.com/paper/2503.09181/) (2025)
- [FRET: Feature Redundancy Elimination for Test Time Adaptation](https://mlsanity.com/paper/2505.10641/) (2025)
- [NOCTA: Non-Greedy Objective Cost-Tradeoff Acquisition for Longitudinal Data](https://mlsanity.com/paper/2507.12412/) (2025)
- [Model-Agnostic Dynamic Feature Selection with Uncertainty Quantification](https://mlsanity.com/paper/2508.02566/) (2025)
- [Non-Myopic Active Feature Acquisition via Pathwise Policy Gradients](https://mlsanity.com/paper/2605.05511/) (2026)

## Abstract (arXiv)

Feature selection helps reduce data acquisition costs in ML, but the standard approach is to train models with static feature subsets. Here, we consider the dynamic feature selection (DFS) problem where a model sequentially queries features based on the presently available information. DFS is often addressed with reinforcement learning, but we explore a simpler approach of greedily selecting features based on their conditional mutual information. This method is theoretically appealing but requires oracle access to the data distribution, so we develop a learning approach based on amortized optimization. The proposed method is shown to recover the greedy policy when trained to optimality, and it outperforms numerous existing feature selection methods in our experiments, thus validating it as a simple but powerful approach for this problem.