mlsanityData to 7 Oct 2026

Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

arXiv 2202.09667 ↗First posted 19 Feb 2022cs.LG math.OC math.STcited by 34

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where online experimentation is limited. However, depending entirely on logged data, OPE/L is sensitive to environment distribution shifts -- discrepancies between the data-generating environment and that where policies are deployed. citetsi2020distributional proposed distributionally robust OPE/L (DROPE/L) to address this, but the proposal relies on inverse-propensity weighting, whose estimation error and regret will deteriorate if propensities are nonparametrically estimated and whose variance is suboptimal even if not. For standard, non-robust, OPE/L, this is solved by doubly robust (DR) methods, but they do not naturally extend to the more complex DROPE/L, which involves a worst-case expectation. In this paper, we propose the first DR algorithms for DROPE/L with KL-divergence uncertainty sets. For evaluation, we propose Localized Doubly Robust DROPE (LDR2OPE) and show that it achieves semiparametric efficiency under weak product rates conditions. Thanks to a localization technique, LDR2OPE only requires fitting a small number of regressions, just like DR methods for standard OPE. For learning, we propose Continuum Doubly Robust DROPL (CDR2OPL) and show that, under a product rate condition involving a continuum of regressions, it enjoys a fast regret rate of Oleft(N-1/2right) even when unknown propensities are nonparametrically estimated. We empirically validate our algorithms in simulations and further extend our results to general f-divergence uncertainty sets.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 24 of its 67 references that are in the map. First posted on arXiv 19 Feb 2022.

Builds on

24 of 67 references.

References · 24

  1. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  2. 2016Generalized Random Forestscited by 309
  3. 2016Variance-based regularization with convex objectivescited by 153
  4. 2019Orthogonal Statistical Learningcited by 150
  5. 2014From Predictive to Prescriptive Analyticscited by 142
  6. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  7. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  8. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  9. 2018Offline Multi-Action Policy Learning: Generalization and Optimizationcited by 64
  10. 2018Orthogonal Random Forest for Causal Inferencecited by 51
  11. 2019Distributionally Robust Reinforcement Learningcited by 40
  12. 2020Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regressioncited by 36
12 more
  1. 2015Counterfactual Risk Minimization: Learning from Logged Bandit Feedbackcited by 33
  2. 2013Program Evaluation and Causal Inference with High-Dimensional Datacited by 29
  3. 2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
  4. 2020Dynamic Batch Learning in High-Dimensional Sparse Linear Contextual Banditscited by 28
  5. 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
  6. 2019Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyondcited by 23
  7. 2021Policy Learning with Adaptively Collected Datacited by 21
  8. 2020Distributionally Robust Batch Contextual Banditscited by 20
  9. 2020Stochastic Optimization Forestscited by 20
  10. 2019Non-Parametric Inference Adaptive to Intrinsic Dimensioncited by 4
  11. 2020Reinforcement Learning-based Product Delivery Frequency Controlcited by 4
  12. 2019Triply Robust Off-Policy Evaluationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 11; 2025: 8; 2026: 5.

34 papers.

Sort:

By area:

  1. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  2. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  3. 2022Online Policy Optimization for Robust MDPcited by 19
  4. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 15
  5. 2023Fair Off-Policy Learning from Observational Datacited by 11
  6. 2023Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Banditscited by 9
  7. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problemcited by 8
  8. 2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
  9. 2025Treatment Effect Estimation for Optimal Decision-Makingcited by 7
  10. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  11. 2024Distributionally Robust Policy Learning under Concept Driftscited by 5
  12. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
22 more
  1. 2024Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Banditscited by 4
  2. 2023Uncertainty-Aware Instance Reweighting for Off-Policy Learningcited by 3
  3. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  4. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  5. 2024Practical Performative Policy Learning with Strategic Agentscited by 3
  6. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  7. 2024Isolated Causal Effects of Natural Languagecited by 2
  8. 2024Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noisecited by 2
  9. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  10. 2023JoinGym: An Efficient Query Optimization Environment for Reinforcement Learningcited by 1
  11. 2023Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programmingcited by 1
  12. 2025Distributionally Robust Optimization with Adversarial Data Contaminationcited by 1
  13. 2025Online Bayesian Risk-Averse Reinforcement Learningcited by 1
  14. 2026Omitted Variable Bias in Language Models Under Distribution Shiftcited by 1
  15. 2026Functional Natural Policy Gradientscited by 1
  16. 2023A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learningcited by 0
  17. 2024Online MDP with Transition Prototypes: A Robust Adaptive Approachcited by 0
  18. 2025Evaluating and Learning Robust Bandit Policies Under Uncertain Causal Mechanismscited by 0
  19. 2025Fast Best-in-Class Regret for Contextual Banditscited by 0
  20. 2026Off-Policy Evaluation with Strategic Agents via Local Disclosurecited by 0
  21. 2026Wasserstein Policy Learning for Distributional Outcomescited by 0
  22. 2026From Unsupervised Subgroups to Hypothetical State-Intervention Policies: An Evaluation of Selected Subgrouping Methods in Observational Health Datacited by 0