# Doubly Robust Policy Evaluation and Optimization

arXiv: https://arxiv.org/abs/1503.02834 · HTML: https://mlsanity.com/paper/1503.02834/

## Key facts

- Cited by 158 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2015.

## Built on by (most cited first, 30 of 158)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Benchmarks for Deep Off-Policy Evaluation](https://mlsanity.com/paper/2103.16596/) (2021)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approach](https://mlsanity.com/paper/2103.14029/) (2021)
- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [Post-Contextual-Bandit Inference](https://mlsanity.com/paper/2106.00418/) (2021)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Minimax Regret Optimization for Robust Machine Learning under Distribution Shift](https://mlsanity.com/paper/2202.05436/) (2022)
- [Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model](https://mlsanity.com/paper/2202.01562/) (2022)
- [Off-Policy Risk Assessment in Contextual Bandits](https://mlsanity.com/paper/2104.08977/) (2021)
- [Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptotics](https://mlsanity.com/paper/2105.03863/) (2021)
- [Conformal Off-Policy Prediction in Contextual Bandits](https://mlsanity.com/paper/2206.04405/) (2022)
- [Policy-Adaptive Estimator Selection for Off-Policy Evaluation](https://mlsanity.com/paper/2211.13904/) (2022)
- [Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality](https://mlsanity.com/paper/2102.11866/) (2021)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank](https://mlsanity.com/paper/2203.17118/) (2022)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction](https://mlsanity.com/paper/2402.02171/) (2024)
- [Optimal Design for Human Preference Elicitation](https://mlsanity.com/paper/2404.13895/) (2024)
- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [Generalizing Off-Policy Learning under Sample Selection Bias](https://mlsanity.com/paper/2112.01387/) (2021)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Safe Exploration for Efficient Policy Evaluation and Comparison](https://mlsanity.com/paper/2202.13234/) (2022)
- [Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketing](https://mlsanity.com/paper/2303.01049/) (2023)
- [Exponential Smoothing for Off-Policy Learning](https://mlsanity.com/paper/2305.15877/) (2023)
- [Reliable Off-Policy Learning for Dosage Combinations](https://mlsanity.com/paper/2305.19742/) (2023)
- [Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits](https://mlsanity.com/paper/2306.07923/) (2023)
- [Off-Policy Evaluation for Large Action Spaces via Policy Convolution](https://mlsanity.com/paper/2310.15433/) (2023)

## Abstract (arXiv)

We study sequential decision making in environments where rewards are only partially observed, but can be modeled as a function of observed contexts and the chosen action by the decision maker. This setting, known as contextual bandits, encompasses a wide variety of applications such as health care, content recommendation and Internet advertising. A central task is evaluation of a new policy given historic data consisting of contexts, actions and received rewards. The key challenge is that the past data typically does not faithfully represent proportions of actions taken by a new policy. Previous approaches rely either on models of rewards or models of the past policy. The former are plagued by a large bias whereas the latter have a large variance. In this work, we leverage the strengths and overcome the weaknesses of the two approaches by applying the doubly robust estimation technique to the problems of policy evaluation and optimization. We prove that this approach yields accurate value estimates when we have either a good (but not necessarily consistent) model of rewards or a good (but not necessarily consistent) model of past policy. Extensive empirical comparison demonstrates that the doubly robust estimation uniformly improves over existing techniques, achieving both lower variance in value estimation and better policies. As such, we expect the doubly robust approach to become common practice in policy evaluation and optimization.