Data to 7 Oct 2026

Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes

arXiv 1908.08526 ↗First posted 22 Aug 2019cs.LG cs.AI stat.MLcited by 118

Not on the map. Why?

Cited by 27 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Off-policy evaluation (OPE) in reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. We consider for the first time the semiparametric efficiency limits of OPE in Markov decision processes (MDPs), where actions, rewards, and states are memoryless. We show existing OPE estimators may fail to be efficient in this setting. We develop a new estimator based on cross-fold estimation of q-functions and marginalized density ratios, which we term double reinforcement learning (DRL). We show that DRL is efficient when both components are estimated at fourth-root rates and is also doubly robust when only one component is consistent. We investigate these properties empirically and demonstrate the performance benefits due to harnessing memorylessness.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 26; 2022: 24; 2023: 17; 2024: 18; 2025: 18; 2026: 12.

118 papers.

Sort:

By area:

2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
2022Neural Causal Models for Counterfactual Identification and Estimationcited by 54
2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
2021Universal Off-Policy Evaluationcited by 43
2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
2021Statistical Inference with M-Estimators on Adaptively Collected Datacited by 40
2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38

38 more

2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
2021Fast Rates for the Regret of Offline Reinforcement Learningcited by 26
2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
2021DoubleML -- An Object-Oriented Implementation of Double Machine Learning in Pythoncited by 23
2022Markovian Interference in Experimentscited by 22
2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
2024Model-agnostic meta-learners for estimating heterogeneous treatment effects over timecited by 18
2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
2023High-probability sample complexities for policy evaluation with linear function approximationcited by 16
2023Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximationcited by 15
2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14
2022Conformal Off-policy Predictioncited by 14
2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
2022Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficientcited by 12
2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
2023Correcting for Interference in Experiments: A Case Study at Douyincited by 12
2025Doubly Robust Alignment for Large Language Modelscited by 11
2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
2024IGC-Net for conditional average potential outcome estimation over timecited by 10
2022Doubly robust nearest neighbors in factor modelscited by 9
2023Marginal Density Ratio for Off-Policy Evaluation in Contextual Banditscited by 9
2023Evaluation of Active Feature Acquisition Methods for Time-varying Feature Settingscited by 9
2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 9
2020Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learningcited by 8

50 most cited of 118 All 118 (JSON)