# Sample Observed Effects: Enumeration, Randomization and Generalization

arXiv: https://arxiv.org/abs/2108.04376 · HTML: https://mlsanity.com/paper/2108.04376/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 65 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 15 of 15)

- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions](https://mlsanity.com/paper/1707.06422/) (2017)
- [Invariance, Causality and Robustness](https://mlsanity.com/paper/1812.08233/) (2018)
- [The Blessings of Multiple Causes](https://mlsanity.com/paper/1805.06826/) (2018)
- [External Validity: From Do-Calculus to Transportability Across Populations](https://mlsanity.com/paper/1503.01603/) (2015)
- [Learning Representations by Maximizing Mutual Information in Variational Autoencoders](https://mlsanity.com/paper/1912.13361/) (2019)
- [Fairness seen as Global Sensitivity Analysis](https://mlsanity.com/paper/2103.04613/) (2021)
- [Causal Inference on Discrete Data via Estimating Distance Correlations](https://mlsanity.com/paper/1803.07712/) (2018)
- [Regression Discontinuity Designs Using Covariates](https://mlsanity.com/paper/1809.03904/) (2018)
- [Na"ive regression requires weaker assumptions than factor models to adjust for multiple cause confounding](https://mlsanity.com/paper/2007.12702/) (2020)
- [What can the millions of random treatments in nonexperimental data reveal about causes?](https://mlsanity.com/paper/2105.01152/) (2021)

## Abstract (arXiv)

The widely used 'Counterfactual' definition of Causal Effects was derived for unbiasedness and accuracy - and not generalizability. We propose a Combinatorial definition for the External Validity (EV) of intervention effects. We first define the concept of an effect observation 'background'. We then formulate conditions for effect generalization based on samples' sets of (observed and unobserved) backgrounds. This reveals two limits for effect generalization: (1) when effects of a variable are observed under all their enumerable backgrounds, or, (2) when backgrounds have become sufficiently randomized. We use the resulting combinatorial framework to re-examine several issues in the original counterfactual formulation: out-of-sample validity, concurrent estimation of multiple effects, bias-variance tradeoffs, statistical power, and connections to current predictive and explaining techniques. Methodologically, the definitions also allow us to replace the parametric estimation problems that followed the counterfactual definition by combinatorial enumeration and randomization problems in non-experimental samples. We use the resulting non-parametric framework to demonstrate (External Validity, Unconfoundness and Precision) tradeoffs in the performance of popular supervised, explaining, and causal-effect estimators. We also illustrate how the approach allows for the use of supervised and explaining methods in non-i.i.d. samples. The COVID19 pandemic highlighted the need for learning solutions to provide predictions in severally incomplete samples. We demonstrate applications in this pressing problem.