mlsanityData to 7 Oct 2026

Goal-Conditioned Reinforcement Learning with Imagined Subgoals

arXiv 2107.00541 ↗First posted 1 Jul 2021cs.LG cs.ROcited by 129

Not on the map. Why?

Cited by 27 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Goal-conditioned reinforcement learning endows an agent with a large variety of skills, but it often struggles to solve tasks that require more temporally extended reasoning. In this work, we propose to incorporate imagined subgoals into policy learning to facilitate learning of complex tasks. Imagined subgoals are predicted by a separate high-level policy, which is trained simultaneously with the policy and its critic. This high-level policy predicts intermediate states halfway to the goal using the value function as a reachability metric. We don't require the policy to reach these subgoals explicitly. Instead, we use them to define a prior policy, and incorporate this prior into a KL-constrained policy iteration scheme to speed up and regularize learning. Imagined subgoals are used during policy learning, but not during test time, where we only apply the learned policy. We evaluate our approach on complex robotic navigation and manipulation tasks and show that it outperforms existing methods by a large margin.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

38 of 52 references.

References · 38

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
26 more
  1. 2020Reinforcement Learning with Augmented Datacited by 452
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  3. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  5. 2018Maximum a Posteriori Policy Optimisationcited by 318
  6. 2020Critic Regularized Regressioncited by 269
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  9. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  11. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  12. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  13. 2019Planning with Goal-Conditioned Policiescited by 144
  14. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  15. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  16. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  17. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  18. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  19. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  20. 2019Information asymmetry in KL-regularized RLcited by 53
  21. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  22. 2018Relative Entropy Regularized Policy Iterationcited by 32
  23. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 27
  24. 2018Many-Goals Reinforcement Learningcited by 22
  25. 2020Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planningcited by 19
  26. 2019Exploiting Hierarchy for Learning and Transfer in KL-regularized RLcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 17; 2023: 25; 2024: 35; 2025: 24; 2026: 24.

129 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  3. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  4. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  5. 2024Foundation Policies with Hilbert Representationscited by 89
  6. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  7. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  8. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  9. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  10. 2023Contrastive Difference Predictive Codingcited by 35
  11. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  12. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
38 more
  1. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  2. 2023Planning Goals for Explorationcited by 28
  3. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  4. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 23
  5. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  6. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  7. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RLcited by 20
  8. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  9. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  10. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  11. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  12. 2022Hierarchical Decision Transformercited by 16
  13. 2021C-Planning: An Automatic Curriculum for Learning Goal-Reaching Taskscited by 15
  14. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  15. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  16. 2025Horizon Generalization in Reinforcement Learningcited by 15
  17. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  18. 2025Efficient Robotic Policy Learning via Latent Space Backward Planningcited by 14
  19. 2022Learning Robotic Navigation from Experience: Principles, Methods, and Recent Resultscited by 13
  20. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  21. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 13
  22. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  23. 2023A Survey of Demonstration Learningcited by 12
  24. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  25. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 12
  26. 2022Automated Play-Testing Through RL Based Human-Like Play-Styles Generationcited by 11
  27. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  28. 2023Learning to Reach Goals via Diffusioncited by 10
  29. 2024Embodied CoT Distillation From LLM To Off-the-shelf Agentscited by 9
  30. 2023Learning to Discover Skills through Guidancecited by 8
  31. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  32. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  33. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  34. 2022Leveraging Sequentiality in Reinforcement Learning from a Single Demonstrationcited by 6
  35. 2023Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigationcited by 6
  36. 2023ReRoGCRL: Representation-based Robustness in Goal-Conditioned Reinforcement Learningcited by 6
  37. 2024Periodic agent-state based Q-learning for POMDPscited by 6
  38. 2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusioncited by 6

50 most cited of 129 All 129 (JSON)