# Testing Generalizability in Causal Inference

arXiv: https://arxiv.org/abs/2411.03021 · HTML: https://mlsanity.com/paper/2411.03021/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Engression: Extrapolation through the Lens of Distributional Regression](https://mlsanity.com/paper/2307.00835/) (2023)
- [Learning Weighted Representations for Generalization Across Designs](https://mlsanity.com/paper/1802.08598/) (2018)
- [Invariant Representation Learning for Treatment Effect Estimation](https://mlsanity.com/paper/2011.12379/) (2020)
- [Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals](https://mlsanity.com/paper/1709.04589/) (2017)
- [Transfer Learning on Heterogeneous Feature Spaces for Treatment Effects Estimation](https://mlsanity.com/paper/2210.06183/) (2022)
- [Estimating Individual Treatment Effects using Non-Parametric Regression Models: a Review](https://mlsanity.com/paper/2009.06472/) (2020)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [Parameterizing and Simulating from Causal Models](https://mlsanity.com/paper/2109.03694/) (2021)
- [Marginal Causal Flows for Validation and Inference](https://mlsanity.com/paper/2411.01295/) (2024)
- [An evaluation framework for comparing causal inference models](https://mlsanity.com/paper/2209.00115/) (2022)

## Abstract (arXiv)

Ensuring robust model performance in diverse real-world scenarios requires addressing generalizability across domains with covariate shifts. However, no formal procedure exists for statistically evaluating generalizability in machine learning algorithms. Existing predictive metrics like mean squared error (MSE) help to quantify the relative performance between models, but do not directly answer whether a model can or cannot generalize. To address this gap in the domain of causal inference, we propose a systematic framework for statistically evaluating the generalizability of high-dimensional causal inference models. Our approach uses the frugal parameterization to flexibly simulate from fully and semi-synthetic causal benchmarks, offering a comprehensive evaluation for both mean and distributional regression methods. Grounded in real-world data, our method ensures more realistic evaluations, which is often missing in current work relying on simplified datasets. Furthermore, using simulations and statistical testing, our framework is robust and avoids over-reliance on conventional metrics, providing statistical safeguards for decision making.