mlsanityData to 7 Oct 2026

Bias Resilient Multi-Step Off-Policy Goal-Conditioned Reinforcement Learning

arXiv 2311.17565 ↗First posted 29 Nov 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In goal-conditioned reinforcement learning (GCRL), sparse rewards present significant challenges, often obstructing efficient learning. Although multi-step GCRL can boost this efficiency, it can also lead to off-policy biases in target values. This paper dives deep into these biases, categorizing them into two distinct categories: "shooting" and "shifting". Recognizing that certain behavior policies can hasten policy refinement, we present solutions designed to capitalize on the positive aspects of these biases while minimizing their drawbacks, enabling the use of larger step sizes to speed up GCRL. An empirical study demonstrates that our approach ensures a resilient and robust improvement, even in ten-step learning scenarios, leading to superior learning efficiency and performance that generally surpass the baseline and several state-of-the-art multi-step GCRL benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

21 of 30 references.

References · 21

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  9. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  10. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  11. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  12. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
9 more
  1. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  2. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  3. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  4. 2016Q(λ) with Off-Policy Correctionscited by 48
  5. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  6. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  7. 2021Unbiased Methods for Multi-Goal Reinforcement Learningcited by 7
  8. 2022USHER: Unbiased Sampling for Hindsight Experience Replaycited by 6
  9. 2021Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learningcited by 4

Built on by

Not cited yet.