# Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning

arXiv: https://arxiv.org/abs/2202.09667 · HTML: https://mlsanity.com/paper/2202.09667/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 67 references that are in the map.
- First posted on arXiv 19 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [From Predictive to Prescriptive Analytics](https://mlsanity.com/paper/1402.5481/) (2014)
- [More Robust Doubly Robust Off-policy Evaluation](https://mlsanity.com/paper/1802.03493/) (2018)
- [Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/1908.08526/) (2019)
- [Sample Complexity of Robust Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2112.01506/) (2021)
- [Offline Multi-Action Policy Learning: Generalization and Optimization](https://mlsanity.com/paper/1810.04778/) (2018)
- [Orthogonal Random Forest for Causal Inference](https://mlsanity.com/paper/1806.03467/) (2018)
- [Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/1902.08708/) (2019)
- [Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regression](https://mlsanity.com/paper/2005.14458/) (2020)
- [Counterfactual Risk Minimization: Learning from Logged Bandit Feedback](https://mlsanity.com/paper/1502.02362/) (2015)
- [Program Evaluation and Causal Inference with High-Dimensional Data](https://mlsanity.com/paper/1311.2645/) (2013)
- [Learning Optimal Distributionally Robust Individualized Treatment Rules](https://mlsanity.com/paper/2006.15121/) (2020)
- [Dynamic Batch Learning in High-Dimensional Sparse Linear Contextual Bandits](https://mlsanity.com/paper/2008.11918/) (2020)
- [Statistically Efficient Off-Policy Policy Gradients](https://mlsanity.com/paper/2002.04014/) (2020)
- [Localized Debiased Machine Learning: Efficient Inference on Quantile Treatment Effects and Beyond](https://mlsanity.com/paper/1912.12945/) (2019)
- [Policy Learning with Adaptively Collected Data](https://mlsanity.com/paper/2105.02344/) (2021)
- [Distributionally Robust Batch Contextual Bandits](https://mlsanity.com/paper/2006.05630/) (2020)
- [Stochastic Optimization Forests](https://mlsanity.com/paper/2008.07473/) (2020)
- [Non-Parametric Inference Adaptive to Intrinsic Dimension](https://mlsanity.com/paper/1901.03719/) (2019)
- [Reinforcement Learning-based Product Delivery Frequency Control](https://mlsanity.com/paper/2012.10858/) (2020)
- [Triply Robust Off-Policy Evaluation](https://mlsanity.com/paper/1911.05811/) (2019)

## Built on by (most cited first, 30 of 34)

- [Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR](https://mlsanity.com/paper/2302.03201/) (2023)
- [The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning](https://mlsanity.com/paper/2305.15703/) (2023)
- [Online Policy Optimization for Robust MDP](https://mlsanity.com/paper/2209.13841/) (2022)
- [Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation](https://mlsanity.com/paper/2411.09891/) (2024)
- [Fair Off-Policy Learning from Observational Data](https://mlsanity.com/paper/2303.08516/) (2023)
- [Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Bandits](https://mlsanity.com/paper/2309.08748/) (2023)
- [Optimizing Language Models for Human Preferences is a Causal Inference Problem](https://mlsanity.com/paper/2402.14979/) (2024)
- [The Central Role of the Loss Function in Reinforcement Learning](https://mlsanity.com/paper/2409.12799/) (2024)
- [Treatment Effect Estimation for Optimal Decision-Making](https://mlsanity.com/paper/2505.13092/) (2025)
- [Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Framework](https://mlsanity.com/paper/2309.13278/) (2023)
- [Distributionally Robust Policy Learning under Concept Drifts](https://mlsanity.com/paper/2412.14297/) (2024)
- [Efficient and Sharp Off-Policy Learning under Unobserved Confounding](https://mlsanity.com/paper/2502.13022/) (2025)
- [Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Bandits](https://mlsanity.com/paper/2401.11353/) (2024)
- [Uncertainty-Aware Instance Reweighting for Off-Policy Learning](https://mlsanity.com/paper/2303.06389/) (2023)
- [Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2402.18159/) (2024)
- [Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes](https://mlsanity.com/paper/2404.00099/) (2024)
- [Practical Performative Policy Learning with Strategic Agents](https://mlsanity.com/paper/2412.01344/) (2024)
- [Reinforcement Learning with Continuous Actions Under Unmeasured Confounding](https://mlsanity.com/paper/2505.00304/) (2025)
- [Isolated Causal Effects of Natural Language](https://mlsanity.com/paper/2410.14812/) (2024)
- [Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise](https://mlsanity.com/paper/2411.06697/) (2024)
- [Towards Optimal Offline Reinforcement Learning](https://mlsanity.com/paper/2503.12283/) (2025)
- [JoinGym: An Efficient Query Optimization Environment for Reinforcement Learning](https://mlsanity.com/paper/2307.11704/) (2023)
- [Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programming](https://mlsanity.com/paper/2310.02459/) (2023)
- [Distributionally Robust Optimization with Adversarial Data Contamination](https://mlsanity.com/paper/2507.10718/) (2025)
- [Online Bayesian Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2509.14077/) (2025)
- [Omitted Variable Bias in Language Models Under Distribution Shift](https://mlsanity.com/paper/2602.16784/) (2026)
- [Functional Natural Policy Gradients](https://mlsanity.com/paper/2603.28681/) (2026)
- [A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learning](https://mlsanity.com/paper/2310.09545/) (2023)
- [Online MDP with Transition Prototypes: A Robust Adaptive Approach](https://mlsanity.com/paper/2412.14075/) (2024)
- [Evaluating and Learning Robust Bandit Policies Under Uncertain Causal Mechanisms](https://mlsanity.com/paper/2508.02812/) (2025)

## Abstract (arXiv)

Off-policy evaluation and learning (OPE/L) use offline observational data to make better decisions, which is crucial in applications where online experimentation is limited. However, depending entirely on logged data, OPE/L is sensitive to environment distribution shifts -- discrepancies between the data-generating environment and that where policies are deployed. citetsi2020distributional proposed distributionally robust OPE/L (DROPE/L) to address this, but the proposal relies on inverse-propensity weighting, whose estimation error and regret will deteriorate if propensities are nonparametrically estimated and whose variance is suboptimal even if not. For standard, non-robust, OPE/L, this is solved by doubly robust (DR) methods, but they do not naturally extend to the more complex DROPE/L, which involves a worst-case expectation. In this paper, we propose the first DR algorithms for DROPE/L with KL-divergence uncertainty sets. For evaluation, we propose Localized Doubly Robust DROPE (LDR2OPE) and show that it achieves semiparametric efficiency under weak product rates conditions. Thanks to a localization technique, LDR2OPE only requires fitting a small number of regressions, just like DR methods for standard OPE. For learning, we propose Continuum Doubly Robust DROPL (CDR2OPL) and show that, under a product rate condition involving a continuum of regressions, it enjoys a fast regret rate of Oleft(N-1/2right) even when unknown propensities are nonparametrically estimated. We empirically validate our algorithms in simulations and further extend our results to general f-divergence uncertainty sets.