# Estimating Conditional Mutual Information for Dynamic Feature Selection

arXiv: https://arxiv.org/abs/2306.03301 · HTML: https://mlsanity.com/paper/2306.03301/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 81 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Concrete Autoencoders for Differentiable Feature Selection and Reconstruction](https://mlsanity.com/paper/1901.09346/) (2019)
- [EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE](https://mlsanity.com/paper/1809.11142/) (2018)
- [VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data](https://mlsanity.com/paper/2006.11941/) (2020)
- [A Petri Dish for Histopathology Image Analysis](https://mlsanity.com/paper/2101.12355/) (2021)
- [Learning to Maximize Mutual Information for Dynamic Feature Selection](https://mlsanity.com/paper/2301.00557/) (2023)
- [Active Feature Acquisition with Generative Surrogate Models](https://mlsanity.com/paper/2010.02433/) (2020)
- [Restricted Strong Convexity Implies Weak Submodularity](https://mlsanity.com/paper/1612.00804/) (2016)
- [Certified Patch Robustness via Smoothed Vision Transformers](https://mlsanity.com/paper/2110.07719/) (2021)
- [Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification](https://mlsanity.com/paper/1711.07592/) (2017)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)
- [Classification with Costly Features using Deep Reinforcement Learning](https://mlsanity.com/paper/1711.07364/) (2017)
- [Interpretable by Design: Learning Predictors by Composing Interpretable Queries](https://mlsanity.com/paper/2207.00938/) (2022)
- [Variational Information Pursuit for Interpretable Predictions](https://mlsanity.com/paper/2302.02876/) (2023)
- [Classification with Costly Features as a Sequential Decision-Making Problem](https://mlsanity.com/paper/1909.02564/) (2019)
- [Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data Streams](https://mlsanity.com/paper/1901.00243/) (2019)
- [BSODA: A Bipartite Scalable Framework for Online Disease Diagnosis](https://mlsanity.com/paper/2012.01065/) (2020)
- [Dropout Feature Ranking for Deep Learning Models](https://mlsanity.com/paper/1712.08645/) (2017)
- [A Probabilistic Hard Attention Model For Sequentially Observed Scenes](https://mlsanity.com/paper/2111.07534/) (2021)

## Built on by (most cited first, 21 of 21)

- [A Survey on Active Feature Acquisition Strategies](https://mlsanity.com/paper/2502.11067/) (2025)
- [Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection](https://mlsanity.com/paper/2602.08003/) (2026)
- [Stochastic Encodings for Active Feature Acquisition](https://mlsanity.com/paper/2508.01957/) (2025)
- [AFABench: A Generic Framework for Benchmarking Active Feature Acquisition](https://mlsanity.com/paper/2508.14734/) (2025)
- [Mutual Information Preserving Neural Network Pruning](https://mlsanity.com/paper/2411.00147/) (2024)
- [Learning-To-Measure: In-Context Active Feature Acquisition](https://mlsanity.com/paper/2510.12624/) (2025)
- [Information Templates: A New Paradigm for Intelligent Active Feature Acquisition](https://mlsanity.com/paper/2508.18380/) (2025)
- [Dynamic Feature Selection from Variable Feature Sets Using Features of Features](https://mlsanity.com/paper/2503.09181/) (2025)
- [NOCTA: Non-Greedy Objective Cost-Tradeoff Acquisition for Longitudinal Data](https://mlsanity.com/paper/2507.12412/) (2025)
- [Model-Agnostic Dynamic Feature Selection with Uncertainty Quantification](https://mlsanity.com/paper/2508.02566/) (2025)
- [Non-Myopic Active Feature Acquisition via Pathwise Policy Gradients](https://mlsanity.com/paper/2605.05511/) (2026)
- [Relaxed Efficient Acquisition of Context and Temporal Features](https://mlsanity.com/paper/2603.11370/) (2026)
- [Hypernetworks for Dynamic Feature Selection](https://mlsanity.com/paper/2605.12278/) (2026)
- [Zero-Shot Active Feature Acquisition via LLM-Elicitation](https://mlsanity.com/paper/2606.18933/) (2026)
- [BRiG-AFA: Bellman Risk-to-Go Learning for Non-Myopic Active Feature Acquisition](https://mlsanity.com/paper/2608.02305/) (2026)
- [Conditional Visual Evidence Utility: State-Dependent Rank Reversals in Frozen Vision-Language Encoders](https://mlsanity.com/paper/2608.28316/) (2026)
- [Explanations-Driven Active Feature Acquisition for Algorithmic Recourse](https://mlsanity.com/paper/2609.12179/) (2026)
- [Learning Source Acquisition Policies by Offline Planning](https://mlsanity.com/paper/2609.14299/) (2026)
- [When Does Synergy Help Active Feature Acquisition? A PID-Based Study](https://mlsanity.com/paper/2609.32301/) (2026)
- [Active Feature Acquisition With Incomplete Training Data](https://mlsanity.com/paper/2609.32325/) (2026)
- [Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition](https://mlsanity.com/paper/2610.03454/) (2026)

## Abstract (arXiv)

Dynamic feature selection, where we sequentially query features to make accurate predictions with a minimal budget, is a promising paradigm to reduce feature acquisition costs and provide transparency into a model's predictions. The problem is challenging, however, as it requires both predicting with arbitrary feature sets and learning a policy to identify valuable selections. Here, we take an information-theoretic perspective and prioritize features based on their mutual information with the response variable. The main challenge is implementing this policy, and we design a new approach that estimates the mutual information in a discriminative rather than generative fashion. Building on our approach, we then introduce several further improvements: allowing variable feature budgets across samples, enabling non-uniform feature costs, incorporating prior information, and exploring modern architectures to handle partial inputs. Our experiments show that our method provides consistent gains over recent methods across a variety of datasets.