mlsanityData to 14 Oct 2026

Planning with Goal-Conditioned Policies

arXiv 1911.08453 ↗First posted 19 Nov 2019cs.LG cs.AI cs.ROcited by 149

Not on the map. Why?

Cited by 32 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, model-free reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but often struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can be incorporated into planning, so that a planner can focus on which states to reach, rather than how those states are reached. However, with complex state observations such as images, not all inputs represent valid states. We therefore also propose using a latent variable model to compactly represent the set of valid states for the planner, so that the policies provide an abstraction of actions, and the latent variable model provides an abstraction of states. We compare our method with planning-based and model-free methods and find that our method significantly outperforms prior work when evaluated on image-based robot navigation and manipulation tasks that require non-greedy, multi-staged behavior.

Key facts. Cited by 149 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 25; 2022: 32; 2023: 25; 2024: 25; 2025: 22; 2026: 19.

149 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  4. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  5. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  6. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  7. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  8. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  9. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  10. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 65
  11. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 57
  12. 2025Horizon Reduction Makes RL Scalablecited by 57
38 more
  1. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 50
  2. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  3. 2022DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Toolscited by 44
  4. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  5. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 43
  6. 2023GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videoscited by 40
  7. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  8. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  9. 2022Offline Reinforcement Learning for Visual Navigationcited by 29
  10. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 28
  11. 2022AnyMorph: Learning Transferable Polices By Inferring Agent Morphologycited by 27
  12. 2021TempoRL: Learning When to Actcited by 25
  13. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 25
  14. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  15. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 25
  16. 2021Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Pathcited by 23
  17. 2021Learning Stochastic Shortest Path with Linear Function Approximationcited by 23
  18. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  19. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 23
  20. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 23
  21. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 23
  22. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  23. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  24. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 19
  25. 2022Humans decompose tasks by trading off utility and computational costcited by 18
  26. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  27. 2021C-Planning: An Automatic Curriculum for Learning Goal-Reaching Taskscited by 17
  28. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  29. 2025Horizon Generalization in Reinforcement Learningcited by 17
  30. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  31. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 15
  32. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  33. 2025Efficient Robotic Policy Learning via Latent Space Backward Planningcited by 14
  34. 2022Learning Robotic Navigation from Experience: Principles, Methods, and Recent Resultscited by 13
  35. 2021Adaptable Agent Populations via a Generative Model of Policiescited by 12
  36. 2022Goal-Space Planning with Subgoal Modelscited by 12
  37. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 11
  38. 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11

50 most cited of 149 All 149 (JSON)