# Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees

arXiv: https://arxiv.org/abs/2204.09938 · HTML: https://mlsanity.com/paper/2204.09938/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 138 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 30 of 34)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game](https://mlsanity.com/paper/2102.13647/) (2021)
- [True to the Model or True to the Data?](https://mlsanity.com/paper/2006.16234/) (2020)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Invariant Representations without Adversarial Training](https://mlsanity.com/paper/1805.09458/) (2018)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [Learning Controllable Fair Representations](https://mlsanity.com/paper/1812.04218/) (2018)
- [Efficient nonparametric statistical inference on population feature importance using Shapley values](https://mlsanity.com/paper/2006.09481/) (2020)
- [Equitability, mutual information, and the maximal information coefficient](https://mlsanity.com/paper/1301.7745/) (2013)
- [Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Models](https://mlsanity.com/paper/1911.04974/) (2019)
- [A simple measure of conditional dependence](https://mlsanity.com/paper/1910.12327/) (2019)
- [A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition](https://mlsanity.com/paper/2105.04187/) (2021)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)
- [SHAFF: Fast and consistent SHApley eFfect estimates via random Forests](https://mlsanity.com/paper/2105.11724/) (2021)
- [Learning Fair Representations for Kernel Models](https://mlsanity.com/paper/1906.11813/) (2019)
- [A Novel Approach to the Partial Information Decomposition](https://mlsanity.com/paper/1908.08642/) (2019)
- [Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data](https://mlsanity.com/paper/2004.08220/) (2020)
- [Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic](https://mlsanity.com/paper/2008.09535/) (2020)
- [Learning Smooth and Fair Representations](https://mlsanity.com/paper/2006.08788/) (2020)
- [Causal Feature Selection via Orthogonal Search](https://mlsanity.com/paper/2007.02938/) (2020)
- [Neural Decomposition: Functional ANOVA with Variational Autoencoders](https://mlsanity.com/paper/2006.14293/) (2020)
- [A Weaker Faithfulness Assumption based on Triple Interactions](https://mlsanity.com/paper/2010.14265/) (2020)
- [Fair Representations by Compression](https://mlsanity.com/paper/2105.14044/) (2021)
- [Greater than the parts: A review of the information decomposition approach to causal emergence](https://mlsanity.com/paper/2111.06518/) (2021)
- [Privacy with Estimation Guarantees](https://mlsanity.com/paper/1710.00447/) (2017)

## Built on by (most cited first, 6 of 6)

- [Partial Information Decomposition for Data Interpretability and Feature Selection](https://mlsanity.com/paper/2405.19212/) (2024)
- [TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning](https://mlsanity.com/paper/2401.11512/) (2024)
- [Feature Selection for Network Intrusion Detection](https://mlsanity.com/paper/2411.11603/) (2024)
- [Stability Selection via Variable Decorrelation](https://mlsanity.com/paper/2505.20864/) (2025)
- [Aggregate Models, Not Explanations: Improving Feature Importance Estimation](https://mlsanity.com/paper/2602.11760/) (2026)
- [Tackling water table depth modeling via machine learning: From proxy observations to verifiability](https://mlsanity.com/paper/2405.04579/) (2024)

## Abstract (arXiv)

Scientists frequently prioritize learning from data rather than training the best possible model; however, research in machine learning often prioritizes the latter. Marginal contribution feature importance (MCI) was developed to break this trend by providing a useful framework for quantifying the relationships in data. In this work, we aim to improve upon the theoretical properties, performance, and runtime of MCI by introducing ultra-marginal feature importance (UMFI), which uses dependence removal techniques from the AI fairness literature as its foundation. We first propose axioms for feature importance methods that seek to explain the causal and associative relationships in data, and we prove that UMFI satisfies these axioms under basic assumptions. We then show on real and simulated data that UMFI performs better than MCI, especially in the presence of correlated interactions and unrelated features, while partially learning the structure of the causal graph and reducing the exponential runtime of MCI to super-linear.