mlsanityData to 14 Oct 2026

Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation

arXiv 2608.24146 ↗First posted 25 Aug 2026cs.LG stat.MLcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not account for uncertainties in the transition functions. In practice, simulator transitions often differ from the real world due to modeling errors or approximation limitations. As a result, behavior policies trained in simulation may still yield high variance when deployed in real environments, leading to costly reliance on real-world evaluation samples. In this work, we propose a double-loop gradient-based algorithm for learning behavior policies that are both efficient and robust to transition uncertainty. Theoretically, we derive novel transition-variance gradient expressions and establish global convergence guarantees for the algorithm. Numerically, we demonstrate that our method is less sensitive to transition perturbations than existing approaches, providing supportive evidence for its practical utility.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 55 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

35 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning29 May
  2. 2024Policy Gradient for Robust Markov Decision Processes29 Oct
  3. 2024Doubly Optimal Policy Evaluation for Reinforcement Learning3 Oct
  4. 2024Efficient Multi-Policy Evaluation for Reinforcement Learning16 Aug
  5. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise15 Jan

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  3. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  4. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  5. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  6. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 50
  7. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  8. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  9. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 37
  10. 2019Active Exploration in Markov Decision Processescited by 27
  11. 2017Data-Efficient Policy Evaluation Through Behavior Policy Searchcited by 26
  12. 2023Robust Average-Reward Markov Decision Processescited by 22
18 more
  1. 2021Minimax Model Learningcited by 16
  2. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 16
  3. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  4. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  5. 2026Beyond Pessimism: Offline Learning in KL-regularized Gamescited by 10
  6. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  7. 2026Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewardscited by 7
  8. 2024Efficient Policy Evaluation with Safety Constraint for Reinforcement Learningcited by 6
  9. 2025Adaptive Exploration for Multi-Reward Multi-Policy Evaluationcited by 6
  10. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 6
  11. 2023Marginalized Importance Sampling for Off-Environment Policy Evaluationcited by 5
  12. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 5
  13. 2026Fast Rates in α-Potential Games via Regularized Mirror Descentcited by 5
  14. 2026Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learningcited by 5
  15. 2026MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematicscited by 4
  16. 2026Predicting Plasticity in Deep Continual Learning: A Theoretical Perspectivecited by 4
  17. 2026Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiationscited by 4
  18. 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classificationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Itemscited by 1
  2. 2026Pessimistic Minimax Learning for Public-Private Information Games under Unilateral Coveragecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning
  2. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  3. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
  4. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  5. 2024Policy Gradient with Active Importance Sampling
  6. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
  7. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  8. 2021Continuous Doubly Constrained Batch Reinforcement Learning