mlsanityData to 7 Oct 2026

Planning Goals for Exploration

arXiv 2303.13002 ↗First posted 23 Mar 2023cs.LG cs.AI cs.ROcited by 28

Not on the map. Why?

Abstract (arXiv)

Dropped into an unknown environment, what should an agent do to quickly learn about the environment and how to accomplish diverse tasks within it? We address this question within the goal-conditioned reinforcement learning paradigm, by identifying how the agent should set its goals at training time to maximize exploration. We propose "Planning Exploratory Goals" (PEG), a method that sets goals for each training episode to directly optimize an intrinsic exploration reward. PEG first chooses goal commands such that the agent's goal-conditioned policy, at its current level of training, will end up in states with high exploration potential. It then launches an exploration policy starting at those promising states. To enable this direct optimization, PEG learns world models and adapts sampling-based planning algorithms to "plan goal commands". In challenging simulated robotics environments including a multi-legged ant robot in a maze, and a robot arm on a cluttered tabletop, PEG exploration enables more efficient and effective training of goal-conditioned policies relative to baselines and ablations. Our ant successfully navigates a long maze, and the robot arm successfully builds a stack of three blocks upon command. Website: https://penn-pal-lab.github.io/peg/

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

43 of 63 references.

References · 43

  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2020Mastering Atari with Discrete World Modelscited by 990
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018DeepMind Control Suitecited by 935
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2022Transformers are Sample-Efficient World Modelscited by 317
31 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  2. 2020First return, then explorecited by 256
  3. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  4. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  5. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  7. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  8. 2017Deep Exploration via Randomized Value Functionscited by 155
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  10. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  11. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  12. 2018Model-Based Active Explorationcited by 105
  13. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  14. 2022Deep Hierarchical Planning from Pixelscited by 96
  15. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  16. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  17. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  18. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  19. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 56
  20. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
  21. 2018Time-Agnostic Prediction: Predicting Predictable Video Framescited by 37
  22. 2021Learning more skills through optimistic explorationcited by 37
  23. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27
  24. 2018Many-Goals Reinforcement Learningcited by 22
  25. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  26. 2021When should agents explore?cited by 20
  27. 2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19
  28. 2021Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reachingcited by 19
  29. 2018Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Modelscited by 11
  30. 2019Curiosity-Driven Multi-Criteria Hindsight Experience Replaycited by 11
  31. 2022When to Go, and When to Explore: The Benefit of Post-Exploration in Intrinsic Motivationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 13; 2025: 6; 2026: 4.

28 papers.

Sort:

By area:

  1. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  2. 2024Simple Hierarchical Planning with Diffusioncited by 71
  3. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  4. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  5. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  6. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 13
  7. 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
  8. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
  9. 2024Dr. Strategy: Model-Based Generalist Agents with Strategic Dreamingcited by 6
  10. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
  11. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  12. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
16 more
  1. 2023Regularity as Intrinsic Reward for Free Playcited by 5
  2. 2025SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Explorationcited by 5
  3. 2023Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreementcited by 4
  4. 2024Reset-free Reinforcement Learning with World Modelscited by 3
  5. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  6. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  7. 2026ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuningcited by 2
  8. 2024Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learningcited by 1
  9. 2025Periodic Skill Discoverycited by 1
  10. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  11. 2024Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learningcited by 0
  12. 2024Multi-Agent Transfer Learning via Temporal Contrastive Learningcited by 0
  13. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  14. 2024Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Taskscited by 0
  15. 2025Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learningcited by 0
  16. 2026Search Inspired Exploration in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
  2. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration
  3. 2019Learning Exploration Policies for Navigation
  4. 2020Guided Exploration with Proximal Policy Optimization using a Single Demonstration
  5. 2026120 Minutes and a Laptop: Minimalist Image-goal Navigation via Unsupervised Exploration and Offline RL
  6. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  7. 2021Discovering and Achieving Goals via World Models
  8. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning