mlsanityData to 7 Oct 2026

Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning

arXiv 2006.03886 ↗First posted 6 Jun 2020cs.LG math.OC stat.MLcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study the efficient off-policy evaluation of natural stochastic policies, which are defined in terms of deviations from the behavior policy. This is a departure from the literature on off-policy evaluation where most work consider the evaluation of explicitly specified policies. Crucially, offline reinforcement learning with natural stochastic policies can help alleviate issues of weak overlap, lead to policies that build upon current practice, and improve policies' implementability in practice. Compared with the classic case of a pre-specified evaluation policy, when evaluating natural stochastic policies, the efficiency bound, which measures the best-achievable estimation error, is inflated since the evaluation policy itself is unknown. In this paper, we derive the efficiency bounds of two major types of natural stochastic policies: tilting policies and modified treatment policies. We then propose efficient nonparametric estimators that attain the efficiency bounds under very lax conditions. These also enjoy a (partial) double robustness property.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 56 references that are in the map. First posted on arXiv 6 Jun 2020.

Builds on

21 of 56 references.

References · 21

  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  6. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  7. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  8. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  9. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  10. 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
  11. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  12. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
9 more
  1. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  2. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  3. 2017Nonparametric causal effects based on incremental propensity score interventionscited by 28
  4. 2019Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learningcited by 28
  5. 2016Assessing Time-Varying Causal Effect Moderation in Mobile Healthcited by 27
  6. 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
  7. 2019More Efficient Off-Policy Evaluation through Regularized Targeted Learningcited by 17
  8. 2020Non-parametric causal effects based on longitudinal modified treatment policiescited by 14
  9. 2019Causal mediation analysis for stochastic interventionscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 0; 2025: 3.

8 papers.

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2023Positivity-free Policy Learning with Observational Datacited by 6
  3. 2022Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learningcited by 2
  4. 2025A Graphical Approach to State Variable Selection in Off-policy Learningcited by 1
  5. 2023Inference for relative sparsitycited by 0
  6. 2023Investigating the Edge of Stability Phenomenon in Reinforcement Learningcited by 0
  7. 2025Treatment, evidence, imitation, and chatcited by 0
  8. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline RL Without Off-Policy Evaluation
  2. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  3. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  7. 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
  8. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data