# Explaining the Behavior of Black-Box Prediction Algorithms with Causal Learning

arXiv: https://arxiv.org/abs/2006.02482 · HTML: https://mlsanity.com/paper/2006.02482/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 10 of 10)

- [D'ya like DAGs? A Survey on Structure Learning and Causal Discovery](https://mlsanity.com/paper/2103.02582/) (2021)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [DiConStruct: Causal Concept-based Explanations through Black-Box Distillation](https://mlsanity.com/paper/2401.08534/) (2024)
- [Causal Dependence Plots](https://mlsanity.com/paper/2303.04209/) (2023)
- [PWSHAP: A Path-Wise Explanation Model for Targeted Variables](https://mlsanity.com/paper/2306.14672/) (2023)
- [I Bet You Did Not Mean That: Testing Semantic Importance via Betting](https://mlsanity.com/paper/2405.19146/) (2024)
- [Trying to Outrun Causality with Machine Learning: Limitations of Model Explainability Techniques for Identifying Predictive Variables](https://mlsanity.com/paper/2202.09875/) (2022)
- [Beyond Single-Feature Importance with ICECREAM](https://mlsanity.com/paper/2307.09779/) (2023)
- [CAPRI-CT: Causal Analysis and Predictive Reasoning for Image Quality Optimization in Computed Tomography](https://mlsanity.com/paper/2507.17420/) (2025)
- [DagSim: Combining DAG-based model structure with unconstrained data types and relations for flexible, transparent, and modularized data simulation](https://mlsanity.com/paper/2205.11234/) (2022)

## Abstract (arXiv)

Causal approaches to post-hoc explainability for black-box prediction models (e.g., deep neural networks trained on image pixel data) have become increasingly popular. However, existing approaches have two important shortcomings: (i) the "explanatory units" are micro-level inputs into the relevant prediction model, e.g., image pixels, rather than interpretable macro-level features that are more useful for understanding how to possibly change the algorithm's behavior, and (ii) existing approaches assume there exists no unmeasured confounding between features and target model predictions, which fails to hold when the explanatory units are macro-level variables. Our focus is on the important setting where the analyst has no access to the inner workings of the target prediction algorithm, rather only the ability to query the output of the model in response to a particular input. To provide causal explanations in such a setting, we propose to learn causal graphical representations that allow for arbitrary unmeasured confounding among features. We demonstrate the resulting graph can differentiate between interpretable features that causally influence model predictions versus those that are merely associated with model predictions due to confounding. Our approach is motivated by a counterfactual theory of causal explanation wherein good explanations point to factors that are "difference-makers" in an interventionist sense.