# A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition

arXiv: https://arxiv.org/abs/2105.04187 · HTML: https://mlsanity.com/paper/2105.04187/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 85 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 6 of 6)

- [Estimating Mutual Information for Discrete-Continuous Mixtures](https://mlsanity.com/paper/1709.06212/) (2017)
- [Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic](https://mlsanity.com/paper/2008.09535/) (2020)
- [JIDT: An information-theoretic toolkit for studying the dynamics of complex systems](https://mlsanity.com/paper/1408.3270/) (2014)
- [Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing](https://mlsanity.com/paper/1902.06828/) (2019)
- [Optimal model-free prediction from multivariate time series](https://mlsanity.com/paper/1506.05822/) (2015)
- [BROJA-2PID: A robust estimator for bivariate partial information decomposition](https://mlsanity.com/paper/1802.02485/) (2018)

## Built on by (most cited first, 26 of 26)

- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts](https://mlsanity.com/paper/2505.19190/) (2025)
- [Quantifying Knowledge Distillation Using Partial Information Decomposition](https://mlsanity.com/paper/2411.07483/) (2024)
- [Demystifying Local and Global Fairness Trade-offs in Federated Learning Using Partial Information Decomposition](https://mlsanity.com/paper/2307.11333/) (2023)
- [MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation](https://mlsanity.com/paper/2508.15281/) (2025)
- [Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks](https://mlsanity.com/paper/2210.02996/) (2022)
- [Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees](https://mlsanity.com/paper/2204.09938/) (2022)
- [Towards Formalizing Spuriousness of Biased Datasets Using Partial Information Decomposition](https://mlsanity.com/paper/2407.00482/) (2024)
- [Partial Information Decomposition for Data Interpretability and Feature Selection](https://mlsanity.com/paper/2405.19212/) (2024)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [A Unified View of Group Fairness Tradeoffs Using Partial Information Decomposition](https://mlsanity.com/paper/2406.04562/) (2024)
- [Feature Selection for Network Intrusion Detection](https://mlsanity.com/paper/2411.11603/) (2024)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Learning Invariant Graph Representations Through Redundant Information](https://mlsanity.com/paper/2512.06154/) (2025)
- [Feature-level Interaction Explanations in Multimodal Transformers](https://mlsanity.com/paper/2603.13326/) (2026)
- [Interaction-Aware Sensitivity Analysis for Aerodynamic Optimization Results using Information Theory](https://mlsanity.com/paper/2112.05609/) (2021)
- [Study Features via Exploring Distribution Structure](https://mlsanity.com/paper/2401.07540/) (2024)
- [Redundancy as a Structural Information Principle for Learning and Generalization](https://mlsanity.com/paper/2510.10938/) (2025)
- [Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control](https://mlsanity.com/paper/2512.24826/) (2025)
- [M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model](https://mlsanity.com/paper/2604.08936/) (2026)
- [Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models](https://mlsanity.com/paper/2605.08145/) (2026)
- [Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?](https://mlsanity.com/paper/2606.29613/) (2026)
- [Partial Information Decomposition as a Multi-Contrast 3D MRI Selection Strategy for Resource-Constrained Deep Neural Network Training in Brain Tumor Segmentation](https://mlsanity.com/paper/2607.15396/) (2026)
- [When Does Synergy Help Active Feature Acquisition? A PID-Based Study](https://mlsanity.com/paper/2609.32301/) (2026)
- [Interpreting Reasoning of Large Language Models via Partial Information Decomposition](https://mlsanity.com/paper/2610.00571/) (2026)
- [Gaussian BROJA Partial Information Decomposition in Closed Form](https://mlsanity.com/paper/2610.13464/) (2026)

## Abstract (arXiv)

Selecting a minimal feature set that is maximally informative about a target variable is a central task in machine learning and statistics. Information theory provides a powerful framework for formulating feature selection algorithms -- yet, a rigorous, information-theoretic definition of feature relevancy, which accounts for feature interactions such as redundant and synergistic contributions, is still missing. We argue that this lack is inherent to classical information theory which does not provide measures to decompose the information a set of variables provides about a target into unique, redundant, and synergistic contributions. Such a decomposition has been introduced only recently by the partial information decomposition (PID) framework. Using PID, we clarify why feature selection is a conceptually difficult problem when approached using information theory and provide a novel definition of feature relevancy and redundancy in PID terms. From this definition, we show that the conditional mutual information (CMI) maximizes relevancy while minimizing redundancy and propose an iterative, CMI-based algorithm for practical feature selection. We demonstrate the power of our CMI-based algorithm in comparison to the unconditional mutual information on benchmark examples and provide corresponding PID estimates to highlight how PID allows to quantify information contribution of features and their interactions in feature-selection problems.