mlsanityData to 14 Oct 2026

Doubly Optimal Policy Evaluation for Reinforcement Learning

arXiv 2410.02226 ↗First posted 3 Oct 2024cs.LGcited by 8

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy evaluation estimates the performance of a policy by (1) collecting data from the environment and (2) processing raw data into a meaningful estimate. Due to the sequential nature of reinforcement learning, any improper data-collecting policy or data-processing method substantially deteriorates the variance of evaluation results over long time steps. Thus, policy evaluation often suffers from large variance and requires massive data to achieve the desired accuracy. In this work, we design an optimal combination of data-collecting policy and data-processing baseline. Theoretically, we prove our doubly optimal policy evaluation method is unbiased and guaranteed to have lower variance than previously best-performing methods. Empirically, compared with previous works, we show our method reduces variance substantially and achieves superior empirical performance.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 45 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

19 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Efficient Multi-Policy Evaluation for Reinforcement Learning16 Aug
  2. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise15 Jan

Other references · 17

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  12. 2017Data-Efficient Policy Evaluation Through Behavior Policy Searchcited by 26
5 more
  1. 2017Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselinescited by 20
  2. 2022ReVar: Strengthening Policy Evaluation via Reduced Variance Samplingcited by 16
  3. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  4. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  5. 2022Semi-analytical Industrial Cooling System Model for Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 6.

8 papers.

  1. 2024Efficient Policy Evaluation with Safety Constraint for Reinforcement Learningcited by 6
  2. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 6
  3. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 5
  4. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 2
  5. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  6. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  7. 2026OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Itemscited by 1
  8. 2026Pessimistic Minimax Learning for Public-Private Information Games under Unilateral Coveragecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning
  2. 2025Evaluation-Aware Reinforcement Learning
  3. 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
  4. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  5. 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning
  6. 2020Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning
  7. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  8. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning