mlsanityData to 13 Oct 2026

GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models

arXiv 2310.20025 ↗First posted 30 Oct 2023cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline Goal-Conditioned RL (GCRL) offers a feasible paradigm for learning general-purpose policies from diverse and multi-task offline datasets. Despite notable recent progress, the predominant offline GCRL methods, mainly model-free, face constraints in handling limited data and generalizing to unseen goals. In this work, we propose Goal-conditioned Offline Planning (GOPlan), a novel model-based framework that contains two key phases: (1) pretraining a prior policy capable of capturing multi-modal action distribution within the multi-goal dataset; (2) employing the reanalysis method with planning to generate imagined trajectories for funetuning policies. Specifically, we base the prior policy on an advantage-weighted conditioned generative adversarial network, which facilitates distinct mode separation, mitigating the pitfalls of out-of-distribution (OOD) actions. For further policy optimization, the reanalysis method generates high-quality imaginary data by planning with learned models for both intra-trajectory and inter-trajectory goals. With thorough experimental evaluations, we demonstrate that GOPlan achieves state-of-the-art performance on various offline multi-goal navigation and manipulation tasks. Moreover, our results highlight the superior ability of GOPlan to handle small data budgets and generalize to OOD goals.

Key facts. Cited by 20 papers in the map (data through 13 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 55 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

45 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?30 May
  2. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning16 Mar
  3. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression7 Jun
  4. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 41

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,782
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,620
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,334
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,241
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,231
  7. 2017Hindsight Experience Replaycited by 1,104
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 884
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 657
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 654
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 637
29 more
  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 632
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 584
  3. 2022Temporal Difference Learning for Model Predictive Controlcited by 384
  4. 2019Learning Latent Plans from Playcited by 313
  5. 2022Behavior Transformers: Cloning k modes with one stonecited by 300
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 298
  7. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 290
  8. 2019Self-Supervised Exploration via Disagreementcited by 264
  9. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  10. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 244
  11. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 201
  12. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  13. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 148
  14. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  15. 2020Model-Based Offline Planningcited by 126
  16. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  17. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  18. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  19. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  20. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  21. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  22. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  23. 2022Can Push-forward Generative Models Fit Multimodal Distributions?cited by 41
  24. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  25. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  26. 2020PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goalscited by 21
  27. 2019A Perspective on Objects and Systematic Generalization in Model-Based RLcited by 14
  28. 2021MHER: Model-based Hindsight Experience Replaycited by 14
  29. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 6; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 224
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 48
  4. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  5. 2025Dual Goal Representationscited by 15
  6. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  7. 2025SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Modelcited by 10
  8. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  9. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
  10. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  11. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  12. 2024Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spacescited by 3
8 more
  1. 2024Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learningcited by 3
  2. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  3. 2025Retrosynthesis Planning via Worst-path Policy Optimisation in Tree-structured MDPscited by 2
  4. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 2
  5. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  6. 2024M3PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Modelcited by 0
  7. 2025Reachability Weighted Offline Goal-conditioned Resamplingcited by 0
  8. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  2. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  3. 2025Flattening Hierarchies with Policy Bootstrapping
  4. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  5. 2024DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learning
  6. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  7. 2024Are Expressive Models Truly Necessary for Offline RL?
  8. 2024Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning