mlsanityData to 7 Oct 2026

On the Reuse Bias in Off-Policy Reinforcement Learning

arXiv 2209.07074 ↗First posted 15 Sep 2022cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Importance sampling (IS) is a popular technique in off-policy evaluation, which re-weights the return of trajectories in the replay buffer to boost sample efficiency. However, training with IS can be unstable and previous attempts to address this issue mainly focus on analyzing the variance of IS. In this paper, we reveal that the instability is also related to a new notion of Reuse Bias of IS -- the bias in off-policy evaluation caused by the reuse of the replay buffer for evaluation and optimization. We theoretically show that the off-policy evaluation and optimization of the current policy with the data from the replay buffer result in an overestimation of the objective, which may cause an erroneous gradient update and degenerate the performance. We further provide a high-probability upper bound of the Reuse Bias, and show that controlling one term of the upper bound can control the Reuse Bias by introducing the concept of stability for off-policy algorithms. Based on these analyses, we finally present a novel Bias-Regularized Importance Sampling (BIRIS) framework along with practical algorithms, which can alleviate the negative impact of the Reuse Bias. Experimental results show that our BIRIS-based methods can significantly improve the sample efficiency on a series of continuous control tasks in MuJoCo.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 58 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

30 of 58 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  12. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
18 more
  1. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  3. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  4. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  5. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  6. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  7. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  8. 2018Policy Optimization via Importance Samplingcited by 47
  9. 2021Universal Off-Policy Evaluationcited by 43
  10. 2018Remember and Forget for Experience Replaycited by 42
  11. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  12. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  13. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  14. 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
  15. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  16. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  17. 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11
  18. 2021High-Confidence Off-Policy (or Counterfactual) Variance Estimationcited by 8

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errorscited by 0
  2. 2024Augmenting Offline RL with Unlabeled Datacited by 0