mlsanityData to 7 Oct 2026

Estimating Conditional Mutual Information for Dynamic Feature Selection

arXiv 2306.03301 ↗First posted 5 Jun 2023cs.LG cs.IT math.ITcited by 20

Not on the map. Why?

Abstract (arXiv)

Dynamic feature selection, where we sequentially query features to make accurate predictions with a minimal budget, is a promising paradigm to reduce feature acquisition costs and provide transparency into a model's predictions. The problem is challenging, however, as it requires both predicting with arbitrary feature sets and learning a policy to identify valuable selections. Here, we take an information-theoretic perspective and prioritize features based on their mutual information with the response variable. The main challenge is implementing this policy, and we design a new approach that estimates the mutual information in a discriminative rather than generative fashion. Building on our approach, we then introduce several further improvements: allowing variable feature budgets across samples, enabling non-uniform feature costs, incorporating prior information, and exploring modern architectures to handle partial inputs. Our experiments show that our method provides consistent gains over recent methods across a variety of datasets.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 34 of its 81 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

34 of 81 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2017Deep Reinforcement Learning that Matterscited by 727
  5. 2018MINE: Mutual Information Neural Estimationcited by 680
  6. 2019On Variational Bounds of Mutual Informationcited by 550
  7. 2021Intriguing Properties of Vision Transformerscited by 446
  8. 2014Recurrent Models of Visual Attentioncited by 344
  9. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  10. 2018Handling Incomplete Heterogeneous Data using VAEscited by 148
  11. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  12. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
22 more
  1. 2019Concrete Autoencoders for Differentiable Feature Selection and Reconstructioncited by 90
  2. 2018EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAEcited by 85
  3. 2020VAEM: a Deep Generative Model for Heterogeneous Mixed Type Datacited by 66
  4. 2021A Petri Dish for Histopathology Image Analysiscited by 63
  5. 2023Learning to Maximize Mutual Information for Dynamic Feature Selectioncited by 54
  6. 2020Active Feature Acquisition with Generative Surrogate Modelscited by 53
  7. 2016Restricted Strong Convexity Implies Weak Submodularitycited by 51
  8. 2021Certified Patch Robustness via Smoothed Vision Transformerscited by 44
  9. 2017Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classificationcited by 39
  10. 2022Missingness Bias in Model Debuggingcited by 36
  11. 2017Classification with Costly Features using Deep Reinforcement Learningcited by 32
  12. 2022Interpretable by Design: Learning Predictors by Composing Interpretable Queriescited by 26
  13. 2023Variational Information Pursuit for Interpretable Predictionscited by 25
  14. 2019Classification with Costly Features as a Sequential Decision-Making Problemcited by 23
  15. 2019Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data Streamscited by 18
  16. 2020BSODA: A Bipartite Scalable Framework for Online Disease Diagnosiscited by 14
  17. 2017Dropout Feature Ranking for Deep Learning Modelscited by 13
  18. 2021A Probabilistic Hard Attention Model For Sequentially Observed Scenescited by 12
  19. 2016Active Information Acquisitioncited by 11
  20. 2015Learning Wake-Sleep Recurrent Attention Modelscited by 9
  21. 2020Neural Joint Entropy Estimationcited by 8
  22. 2021Dynamic Instance-Wise Classification in Correlated Feature Spacescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 11.

20 papers.

Sort:

  1. 2025A Survey on Active Feature Acquisition Strategiescited by 16
  2. 2026Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selectioncited by 10
  3. 2025Stochastic Encodings for Active Feature Acquisitioncited by 8
  4. 2025AFABench: A Generic Framework for Benchmarking Active Feature Acquisitioncited by 7
  5. 2024Mutual Information Preserving Neural Network Pruningcited by 5
  6. 2025Learning-To-Measure: In-Context Active Feature Acquisitioncited by 4
  7. 2025Information Templates: A New Paradigm for Intelligent Active Feature Acquisitioncited by 2
  8. 2025Dynamic Feature Selection from Variable Feature Sets Using Features of Featurescited by 1
  9. 2025NOCTA: Non-Greedy Objective Cost-Tradeoff Acquisition for Longitudinal Datacited by 1
  10. 2025Model-Agnostic Dynamic Feature Selection with Uncertainty Quantificationcited by 1
  11. 2026Non-Myopic Active Feature Acquisition via Pathwise Policy Gradientscited by 1
  12. 2026Relaxed Efficient Acquisition of Context and Temporal Featurescited by 0
8 more
  1. 2026Hypernetworks for Dynamic Feature Selectioncited by 0
  2. 2026Zero-Shot Active Feature Acquisition via LLM-Elicitationcited by 0
  3. 2026BRiG-AFA: Bellman Risk-to-Go Learning for Non-Myopic Active Feature Acquisitioncited by 0
  4. 2026Conditional Visual Evidence Utility: State-Dependent Rank Reversals in Frozen Vision-Language Encoderscited by 0
  5. 2026Explanations-Driven Active Feature Acquisition for Algorithmic Recoursecited by 0
  6. 2026Learning Source Acquisition Policies by Offline Planningcited by 0
  7. 2026When Does Synergy Help Active Feature Acquisition? A PID-Based Studycited by 0
  8. 2026Active Feature Acquisition With Incomplete Training Datacited by 0