# Bayesian Counterfactual Mean Embeddings and Off-Policy Evaluation

arXiv: https://arxiv.org/abs/2211.01518 · HTML: https://mlsanity.com/paper/2211.01518/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons](https://mlsanity.com/paper/2201.07766/) (2022)
- [Kernel Instrumental Variable Regression](https://mlsanity.com/paper/1906.00232/) (2019)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Semiparametric counterfactual density estimation](https://mlsanity.com/paper/2102.12034/) (2021)
- [Counterfactual Mean Embeddings](https://mlsanity.com/paper/1805.08845/) (2018)
- [Off-Policy Risk Assessment in Contextual Bandits](https://mlsanity.com/paper/2104.08977/) (2021)
- [Off-Policy Evaluation and Learning for External Validity under a Covariate Shift](https://mlsanity.com/paper/2002.11642/) (2020)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Deconditional Downscaling with Gaussian Processes](https://mlsanity.com/paper/2105.12909/) (2021)
- [BayesIMP: Uncertainty Quantification for Causal Data Fusion](https://mlsanity.com/paper/2106.03477/) (2021)
- [Bayesian Learning of Kernel Embeddings](https://mlsanity.com/paper/1603.02160/) (2016)
- [Off-policy Learning with Eligibility Traces: A Survey](https://mlsanity.com/paper/1304.3999/) (2013)
- [Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry](https://mlsanity.com/paper/2010.00110/) (2020)
- [Bayesian Kernel Two-Sample Testing](https://mlsanity.com/paper/2002.05550/) (2020)
- [Sequential Decision Making on Unmatched Data using Bayesian Kernel Embeddings](https://mlsanity.com/paper/2210.13692/) (2022)

## Abstract (arXiv)

The counterfactual distribution models the effect of the treatment in the untreated group. While most of the work focuses on the expected values of the treatment effect, one may be interested in the whole counterfactual distribution or other quantities associated to it. Building on the framework of Bayesian conditional mean embeddings, we propose a Bayesian approach for modeling the counterfactual distribution, which leads to quantifying the epistemic uncertainty about the distribution. The framework naturally extends to the setting where one observes multiple treatment effects (e.g. an intermediate effect after an interim period, and an ultimate treatment effect which is of main interest) and allows for additionally modelling uncertainty about the relationship of these effects. For such goal, we present three novel Bayesian methods to estimate the expectation of the ultimate treatment effect, when only noisy samples of the dependence between intermediate and ultimate effects are provided. These methods differ on the source of uncertainty considered and allow for combining two sources of data. Moreover, we generalize these ideas to the off-policy evaluation framework, which can be seen as an extension of the counterfactual estimation problem. We empirically explore the calibration of the algorithms in two different experimental settings which require data fusion, and illustrate the value of considering the uncertainty stemming from the two sources of data.