# High-Confidence Off-Policy (or Counterfactual) Variance Estimation

arXiv: https://arxiv.org/abs/2101.09847 · HTML: https://mlsanity.com/paper/2101.09847/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 61 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling](https://mlsanity.com/paper/1906.03393/) (2019)
- [Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting](https://mlsanity.com/paper/2006.10460/) (2020)
- [Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation](https://mlsanity.com/paper/2007.13609/) (2020)
- [A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning](https://mlsanity.com/paper/1607.00446/) (2016)
- [Conditional Importance Sampling for Off-Policy Learning](https://mlsanity.com/paper/1910.07479/) (2019)
- [Variance Adjusted Actor Critic Algorithms](https://mlsanity.com/paper/1310.3697/) (2013)

## Built on by (most cited first, 8 of 8)

- [Universal Off-Policy Evaluation](https://mlsanity.com/paper/2104.12820/) (2021)
- [Off-Policy Risk Assessment in Contextual Bandits](https://mlsanity.com/paper/2104.08977/) (2021)
- [Off-Policy Risk Assessment in Markov Decision Processes](https://mlsanity.com/paper/2209.10444/) (2022)
- [Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2402.07598/) (2024)
- [Supervised Learning with General Risk Functionals](https://mlsanity.com/paper/2206.13648/) (2022)
- [On the Reuse Bias in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2209.07074/) (2022)
- [Optimal Policy Learning with Observational Data in Multi-Action Scenarios: Estimation, Risk Preference, and Potential Failures](https://mlsanity.com/paper/2403.20250/) (2024)
- [When Does Action Credit Need Updating?](https://mlsanity.com/paper/2609.29007/) (2026)

## Abstract (arXiv)

Many sequential decision-making systems leverage data collected using prior policies to propose a new policy. For critical applications, it is important that high-confidence guarantees on the new policy's behavior are provided before deployment, to ensure that the policy will behave as desired. Prior works have studied high-confidence off-policy estimation of the expected return, however, high-confidence off-policy estimation of the variance of returns can be equally critical for high-risk applications. In this paper, we tackle the previously open problem of estimating and bounding, with high confidence, the variance of returns from off-policy data