# Reliable Off-policy Evaluation for Reinforcement Learning

arXiv: https://arxiv.org/abs/2011.04102 · HTML: https://mlsanity.com/paper/2011.04102/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Sinkhorn Distributionally Robust Optimization](https://mlsanity.com/paper/2109.11926/) (2021)
- [Model Based Reinforcement Learning for Personalized Heparin Dosing](https://mlsanity.com/paper/2304.10000/) (2023)
- [Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms](https://mlsanity.com/paper/2305.07248/) (2023)
- [Offline Policy Comparison under Limited Historical Agent-Environment Interactions](https://mlsanity.com/paper/2106.03934/) (2021)
- [Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes](https://mlsanity.com/paper/2404.00099/) (2024)
- [Towards Optimal Offline Reinforcement Learning](https://mlsanity.com/paper/2503.12283/) (2025)
- [Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies](https://mlsanity.com/paper/2212.08131/) (2022)
- [Iterative Sampling Methods for Sinkhorn Distributionally Robust Optimization](https://mlsanity.com/paper/2512.12550/) (2025)
- [Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation](https://mlsanity.com/paper/2604.18972/) (2026)

## Abstract (arXiv)

In a sequential decision-making problem, off-policy evaluation estimates the expected cumulative reward of a target policy using logged trajectory data generated from a different behavior policy, without execution of the target policy. Reinforcement learning in high-stake environments, such as healthcare and education, is often limited to off-policy settings due to safety or ethical concerns, or inability of exploration. Hence it is imperative to quantify the uncertainty of the off-policy estimate before deployment of the target policy. In this paper, we propose a novel framework that provides robust and optimistic cumulative reward estimates using one or multiple logged trajectories data. Leveraging methodologies from distributionally robust optimization, we show that with proper selection of the size of the distributional uncertainty set, these estimates serve as confidence bounds with non-asymptotic and asymptotic guarantees under stochastic or adversarial environments. Our results are also generalized to batch reinforcement learning and are supported by empirical analysis.