mlsanityData to 14 Oct 2026

Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation

arXiv 2609.19414 ↗First posted 16 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Goal-conditioned reinforcement learning aims to learn policies that reach specified goals, but remains challenging in offline settings with sparse rewards and long-horizon dependencies. In such settings, goal-completion information can be temporally distant from the early decisions that enable success, while offline value estimation introduces additional error. We study this issue from a reward-propagation perspective and show, in a stylized delayed-goal setting, how goal-directed value separation can become small relative to local estimation error. Motivated by this analysis, we propose Reward Stimulation Implicit Q-Learning (RSIQL), a simple non-hierarchical method that introduces additional reward signals at progress-making intermediate states in offline trajectories. RSIQL uses an auxiliary goal-conditioned value function to identify intermediate states estimated to make progress toward the goal and applies reward stimulation to provide less-delayed training supervision. Unlike hierarchical methods, RSIQL does not learn a separate high-level subgoal policy. Experiments on D4RL goal-reaching benchmarks and OGBench show that RSIQL improves over goal-conditioned IQL on average and achieves performance competitive with hierarchical offline goal-conditioned methods, while retaining a simple flat policy structure.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 75 of its 101 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

75 of 101 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning19 May
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  3. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learning7 Jul
  4. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?30 May
  5. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug
  7. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 68

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
38 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  5. 2019Model-Based Reinforcement Learning for Ataricited by 502
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  9. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  10. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  11. 2019Learning Latent Plans from Playcited by 316
  12. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  13. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  14. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  15. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  16. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  17. 2017Stochastic Variational Video Predictioncited by 234
  18. 2018Semi-parametric Topological Memory for Navigationcited by 209
  19. 2020Neural Topological SLAM for Visual Navigationcited by 195
  20. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  21. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  22. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  23. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  24. 2019Goal-conditioned Imitation Learningcited by 162
  25. 2019Planning with Goal-Conditioned Policiescited by 149
  26. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  27. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  28. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  29. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  30. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  31. 2018Count-Based Exploration with the Successor Representationcited by 112
  32. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  33. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  34. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  35. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  36. 2018Motion Planning Networkscited by 80
  37. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 78
  38. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 69

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Goal-Conditioned Q-Learning as Knowledge Distillation
  2. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  3. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
  4. 2026Offline RL with Hierarchical Action Chunking
  5. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning
  6. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  7. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression
  8. 2025Flattening Hierarchies with Policy Bootstrapping