# Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variables

arXiv: https://arxiv.org/abs/2202.09875 · HTML: https://mlsanity.com/paper/2202.09875/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 20 Feb 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [On the Fairness of Disentangled Representations](https://mlsanity.com/paper/1905.13662/) (2019)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [True to the Model or True to the Data?](https://mlsanity.com/paper/2006.16234/) (2020)
- [Drug discovery with explainable artificial intelligence](https://mlsanity.com/paper/2007.00523/) (2020)
- [Does Gender Matter? Towards Fairness in Dialogue Systems](https://mlsanity.com/paper/1910.10486/) (2019)
- [Causal Structure Learning](https://mlsanity.com/paper/1706.09141/) (2017)
- [Hyperparameters and Tuning Strategies for Random Forest](https://mlsanity.com/paper/1804.03515/) (2018)
- [Toward Gender-Inclusive Coreference Resolution](https://mlsanity.com/paper/1910.13913/) (2019)
- [Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information](https://mlsanity.com/paper/1709.01447/) (2017)
- [Treatment effect estimation with disentangled latent factors](https://mlsanity.com/paper/2001.10652/) (2020)
- [Structural Agnostic Modeling: Adversarial Learning of Causal Graphs](https://mlsanity.com/paper/1803.04929/) (2018)
- [Unsuitability of NOTEARS for Causal Graph Discovery](https://mlsanity.com/paper/2104.05441/) (2021)
- [Efficient Estimation of Mutual Information for Strongly Dependent Variables](https://mlsanity.com/paper/1411.2003/) (2014)
- [Equitability, mutual information, and the maximal information coefficient](https://mlsanity.com/paper/1301.7745/) (2013)
- [Estimating Mutual Information for Discrete-Continuous Mixtures](https://mlsanity.com/paper/1709.06212/) (2017)
- [A Constraint-Based Algorithm For Causal Discovery with Cycles, Latent Variables and Selection Bias](https://mlsanity.com/paper/1805.02087/) (2018)
- [Explaining the Behavior of Black-Box Prediction Algorithms with Causal Learning](https://mlsanity.com/paper/2006.02482/) (2020)
- [Intact-VAE: Estimating Treatment Effects under Unobserved Confounding](https://mlsanity.com/paper/2101.06662/) (2021)
- [Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations](https://mlsanity.com/paper/1411.4342/) (2014)
- [Shadow-Mapping for Unsupervised Neural Causal Discovery](https://mlsanity.com/paper/2104.08183/) (2021)

## Built on by (most cited first, 2 of 2)

- [CaTs and DAGs: Integrating Directed Acyclic Graphs with Transformers for Causally Constrained Predictions](https://mlsanity.com/paper/2410.14485/) (2024)
- [WISCA: A Consensus-Based Approach to Harmonizing Interpretability in Tabular Datasets](https://mlsanity.com/paper/2506.06455/) (2025)

## Abstract (arXiv)

Machine Learning explainability techniques have been proposed as a means of `explaining' or interrogating a model in order to understand why a particular decision or prediction has been made. Such an ability is especially important at a time when machine learning is being used to automate decision processes which concern sensitive factors and legal outcomes. Indeed, it is even a requirement according to EU law. Furthermore, researchers concerned with imposing overly restrictive functional form (e.g., as would be the case in a linear regression) may be motivated to use machine learning algorithms in conjunction with explainability techniques, as part of exploratory research, with the goal of identifying important variables which are associated with an outcome of interest. For example, epidemiologists might be interested in identifying `risk factors' - i.e. factors which affect recovery from disease - by using random forests and assessing variable relevance using importance measures. However, and as we demonstrate, machine learning algorithms are not as flexible as they might seem, and are instead incredibly sensitive to the underling causal structure in the data. The consequences of this are that predictors which are, in fact, critical to a causal system and highly correlated with the outcome, may nonetheless be deemed by explainability techniques to be unrelated/unimportant/unpredictive of the outcome. Rather than this being a limitation of explainability techniques per se, we show that it is rather a consequence of the mathematical implications of regression, and the interaction of these implications with the associated conditional independencies of the underlying causal structure. We provide some alternative recommendations for researchers wanting to explore the data for important variables.