# Image-based Treatment Effect Heterogeneity

arXiv: https://arxiv.org/abs/2206.06417 · HTML: https://mlsanity.com/paper/2206.06417/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 60 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 30 of 31)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Recursive Partitioning for Heterogeneous Causal Effects](https://mlsanity.com/paper/1504.01132/) (2015)
- [Deep Structural Causal Models for Tractable Counterfactual Inference](https://mlsanity.com/paper/2006.06485/) (2020)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches](https://mlsanity.com/paper/1803.04386/) (2018)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Using satellite imagery to understand and promote sustainable development](https://mlsanity.com/paper/2010.06988/) (2020)
- [Visual Causal Feature Learning](https://mlsanity.com/paper/1412.2309/) (2014)
- [Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language](https://mlsanity.com/paper/2110.15358/) (2021)
- [DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training](https://mlsanity.com/paper/1802.05664/) (2018)
- [Causal Effect Inference for Structured Treatments](https://mlsanity.com/paper/2106.01939/) (2021)
- [Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes](https://mlsanity.com/paper/1906.05323/) (2019)
- [Multi-Level Cause-Effect Systems](https://mlsanity.com/paper/1512.07942/) (2015)
- [Unsupervised Discovery of El Nino Using Causal Feature Learning on Microlevel Climate Data](https://mlsanity.com/paper/1605.09370/) (2016)
- [Regularizing towards Causal Invariance: Linear Models with Proxies](https://mlsanity.com/paper/2103.02477/) (2021)
- [Statistical modeling: the three cultures](https://mlsanity.com/paper/2012.04570/) (2020)
- [Personalized Public Policy Analysis in Social Sciences using Causal-Graphical Normalizing Flows](https://mlsanity.com/paper/2202.03281/) (2022)
- [Selective inference for effect modification via the lasso](https://mlsanity.com/paper/1705.08020/) (2017)
- [A unified view of likelihood ratio and reparameterization gradients](https://mlsanity.com/paper/2105.14900/) (2021)
- [Randomization Inference for Treatment Effect Variation](https://mlsanity.com/paper/1412.5000/) (2014)
- [Estimating Heterogeneous Causal Effects of High-Dimensional Treatments: Application to Conjoint Analysis](https://mlsanity.com/paper/2201.01357/) (2022)
- [Causal Inference in the Presence of Interference in Sponsored Search Advertising](https://mlsanity.com/paper/2010.07458/) (2020)

## Built on by (most cited first, 20 of 20)

- [Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatments](https://mlsanity.com/paper/2410.00903/) (2024)
- [Causal Effect Estimation: Recent Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2302.00848/) (2023)
- [Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference](https://mlsanity.com/paper/2401.01426/) (2024)
- [Prediction-Powered Causal Inferences](https://mlsanity.com/paper/2502.06343/) (2025)
- [Integrating Earth Observation Data into Causal Inference: Challenges and Opportunities](https://mlsanity.com/paper/2301.12985/) (2023)
- [A Scoping Review of Earth Observation and Machine Learning for Causal Inference: Implications for the Geography of Poverty](https://mlsanity.com/paper/2406.02584/) (2024)
- [Optimizing Multi-Scale Representations to Detect Effect Heterogeneity Using Earth Observation and Computer Vision: Applications to Two Anti-Poverty RCTs](https://mlsanity.com/paper/2411.02134/) (2024)
- [Adjustment for Confounding using Pre-Trained Representations](https://mlsanity.com/paper/2506.14329/) (2025)
- [SSL Framework for Causal Inconsistency between Structures and Representations](https://mlsanity.com/paper/2310.18634/) (2023)
- [DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimation](https://mlsanity.com/paper/2406.00611/) (2024)
- [Debiasing Machine Learning Predictions for Causal Inference Without Additional Ground Truth Data: "One Map, Many Trials" in Satellite-Driven Poverty Analysis](https://mlsanity.com/paper/2508.01341/) (2025)
- [CausalImages: An R Package for Causal Inference with Earth Observation, Bio-medical, and Social Science Images](https://mlsanity.com/paper/2310.00233/) (2023)
- [ReLU integral probability metric and its applications](https://mlsanity.com/paper/2504.18897/) (2025)
- [Towards Causal Relationship in Indefinite Data: Baseline Model and New Datasets](https://mlsanity.com/paper/2401.08221/) (2024)
- [Enhancing predictive imaging biomarker discovery through treatment effect analysis](https://mlsanity.com/paper/2406.02534/) (2024)
- [Compositional Models for Estimating Causal Effects](https://mlsanity.com/paper/2406.17714/) (2024)
- [Annotation-Assisted Learning of Treatment Policies From Multimodal Electronic Health Records](https://mlsanity.com/paper/2507.20993/) (2025)
- [Detecting and Mitigating Treatment Leakage in Text-Based Causal Inference: Distillation and Sensitivity Analysis](https://mlsanity.com/paper/2601.02400/) (2025)
- [Beyond Interaction Effects: Two Logics for Studying Population Inequalities](https://mlsanity.com/paper/2601.04223/) (2025)
- [From Tokens to Policy: Causal and Interpretable Heterogeneous Treatment Effects Identification](https://mlsanity.com/paper/2606.17010/) (2026)

## Abstract (arXiv)

Randomized controlled trials (RCTs) are considered the gold standard for estimating the average treatment effect (ATE) of interventions. One use of RCTs is to study the causes of global poverty -- a subject explicitly cited in the 2019 Nobel Memorial Prize awarded to Duflo, Banerjee, and Kremer "for their experimental approach to alleviating global poverty." Because the ATE is a population summary, anti-poverty experiments often seek to unpack the effect variation around the ATE by conditioning (CATE) on tabular variables such as age and ethnicity that were measured during the RCT data collection. Although such variables are key to unpacking CATE, using only such variables may fail to capture historical, geographical, or neighborhood-specific contributors to effect variation, as tabular RCT data are often only observed near the time of the experiment. In global poverty research, when the location of the experiment units is approximately known, satellite imagery can provide a window into such factors important for understanding heterogeneity. However, there is no method that specifically enables applied researchers to analyze CATE from images. In this paper, using a deep probabilistic modeling framework, we develop such a method that estimates latent clusters of images by identifying images with similar treatment effects distributions. Our interpretable image CATE model also includes a sensitivity factor that quantifies the importance of image segments contributing to the effect cluster prediction. We compare the proposed methods against alternatives in simulation; also, we show how the model works in an actual RCT, estimating the effects of an anti-poverty intervention in northern Uganda and obtaining a posterior predictive distribution over effects for the rest of the country where no experimental data was collected. We make all models available in open-source software.