mlsanityData to 7 Oct 2026

More Robust Doubly Robust Off-policy Evaluation

arXiv 1802.03493 ↗First posted 10 Feb 2018cs.AIcited by 141

Not on the map. Why?

Cited by 24 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where the goal is to estimate the performance of a policy from the data generated by another policy(ies). In particular, we focus on the doubly robust (DR) estimators that consist of an importance sampling (IS) component and a performance model, and utilize the low (or zero) bias of IS and low variance of the model at the same time. Although the accuracy of the model has a huge impact on the overall performance of DR, most of the work on using the DR estimators in OPE has been focused on improving the IS part, and not much on how to learn the model. In this paper, we propose alternative DR estimators, called more robust doubly robust (MRDR), that learn the model parameter by minimizing the variance of the DR estimator. We first present a formulation for learning the DR model in RL. We then derive formulas for the variance of the DR estimator in both contextual bandits and RL, such that their gradients w.r.t.~the model parameters can be estimated from the samples, and propose methods to efficiently minimize the variance. We prove that the MRDR estimators are strongly consistent and asymptotically optimal. Finally, we evaluate MRDR in bandits and RL benchmark problems, and compare its performance with the existing methods.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 3; 2020: 6; 2021: 28; 2022: 23; 2023: 24; 2024: 28; 2025: 17; 2026: 12.

141 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  4. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  5. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  7. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  8. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  9. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  10. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  11. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  12. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
38 more
  1. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  2. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  3. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  4. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  5. 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
  6. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  7. 2021Post-Contextual-Bandit Inferencecited by 32
  8. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  9. 2022Minimax Regret Optimization for Robust Machine Learning under Distribution Shiftcited by 31
  10. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  11. 2021Enhanced Doubly Robust Learning for Debiasing Post-click Conversion Rate Estimationcited by 28
  12. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  13. 2021Active Offline Policy Selectioncited by 26
  14. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  15. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  16. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  17. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  18. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  19. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  20. 2022PAC-Bayesian Offline Contextual Bandits With Guaranteescited by 19
  21. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  22. 2024Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstractioncited by 17
  23. 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
  24. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  25. 2023Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketingcited by 15
  26. 2023Exponential Smoothing for Off-Policy Learningcited by 15
  27. 2022Conformal Off-policy Predictioncited by 14
  28. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  29. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  30. 2020Self-Imitation Learning via Generalized Lower Bound Q-learningcited by 13
  31. 2021Control Variates for Slate Off-Policy Evaluationcited by 13
  32. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  33. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
  34. 2024Optimal Baseline Corrections for Off-Policy Contextual Banditscited by 13
  35. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  36. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 12
  37. 2023A Survey of Demonstration Learningcited by 12
  38. 2023Ad-load Balancing via Off-policy Learning in a Content Marketplacecited by 11

50 most cited of 141 All 141 (JSON)