mlsanityData to 7 Oct 2026

Swapped goal-conditioned offline reinforcement learning

arXiv 2302.08865 ↗First posted 17 Feb 2023cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline goal-conditioned reinforcement learning (GCRL) can be challenging due to overfitting to the given dataset. To generalize agents' skills outside the given dataset, we propose a goal-swapping procedure that generates additional trajectories. To alleviate the problem of noise and extrapolation errors, we present a general offline reinforcement learning method called deterministic Q-advantage policy gradient (DQAPG). In the experiments, DQAPG outperforms state-of-the-art goal-conditioned offline RL methods in a wide range of benchmark tasks, and goal-swapping further improves the test results. It is noteworthy, that the proposed method obtains good performance on the challenging dexterous in-hand manipulation tasks for which the prior methods failed.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 37 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

28 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 27

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
15 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  3. 2020Critic Regularized Regressioncited by 269
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  7. 2021Offline RL Without Off-Policy Evaluationcited by 150
  8. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  10. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  11. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  12. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  13. 2021On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learningcited by 35
  14. 2022Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learningcited by 12
  15. 2019Improving Deep Reinforcement Learning in Minecraft with Action Advicecited by 10

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5
  2. 2024MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learningcited by 1
  3. 2025Prompting Decision Transformers for Zero-Shot Reach-Avoid Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression
  2. 2023Prioritized offline Goal-swapping Experience Replay
  3. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  4. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  5. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
  6. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
  7. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  8. 2025Test-time Offline Reinforcement Learning on Goal-related Experience