Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes
Not on the map. Why?
Abstract (arXiv)
We study the evaluation of a policy under best- and worst-case perturbations to a Markov decision process (MDP), using transition observations from the original MDP, whether they are generated under the same or a different policy. This is an important problem when there is the possibility of a shift between historical and future environments, textite.g. due to unmeasured confounding, distributional shift, or an adversarial environment. We propose a perturbation model that allows changes in the transition kernel densities up to a given multiplicative factor or its reciprocal, extending the classic marginal sensitivity model (MSM) for single time-step decision-making to infinite-horizon RL. We characterize the sharp bounds on policy value under this model unicodex2013 textiti.e., the tightest possible bounds based on transition observations from the original MDP unicodex2013 and we study the estimation of these bounds from such transition observations. We develop an estimator with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates. Our estimator is also asymptotically normal, enabling straightforward statistical inference using Wald confidence intervals. Moreover, when certain nuisances are estimated inconsistently, the estimator still provides valid, albeit possibly not sharp, bounds on the policy value. We validate these properties in numerical simulations. The combination of accounting for environment shifts from train to test (robustness), being insensitive to nuisance-function estimation (orthogonality), and addressing the challenge of learning from finite samples (inference) together leads to credible and reliable policy evaluation.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 45 of its 81 references that are in the map. First posted on arXiv 29 Mar 2024.
Builds on
45 of 81 references.
References · 45
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2019Orthogonal Statistical Learningcited by 150
- 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
- 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
- 2022Robust Reinforcement Learning using Offline Datacited by 95
- 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
- 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
33 more
- 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
- 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
- 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
- 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
- 2015The Influence Function of Semiparametric Estimatorscited by 36
- 2020Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regressioncited by 36
- 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
- 2021Sharp Sensitivity Analysis for Inverse Propensity Weighting via Quantile Balancingcited by 32
- 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
- 2013Program Evaluation and Causal Inference with High-Dimensional Datacited by 29
- 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
- 2017Nonparametric causal effects based on incremental propensity score interventionscited by 28
- 2021Doubly-Valid/Doubly-Sharp Sensitivity Analysis for Causal Inference with Unmeasured Confoundingcited by 28
- 2023B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Confoundingcited by 28
- 2020Double/Debiased Machine Learning for Dynamic Treatment Effects via g-Estimationcited by 27
- 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
- 2021Long Story Short: Omitted Variable Bias in Causal Machine Learningcited by 23
- 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
- 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
- 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
- 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
- 2024Harnessing Density Ratios for Online Reinforcement Learningcited by 15
- 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
- 2022What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatmentcited by 14
- 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 14
- 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
- 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
- 2022Sensitivity Analysis for Marginal Structural Modelscited by 10
- 2020Reliable Off-policy Evaluation for Reinforcement Learningcited by 9
- 2015On the Dual Representation of Coherent Risk Measurescited by 8
- 2024Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learningcited by 8
- 2023Low-Rank MDPs with Continuous Action Spacescited by 4
- 2019Sepsis World Model: A MIMIC-based OpenAI Gym "World Model" Simulator for Sepsis Treatmentcited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.