Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Exploring unknown environments efficiently is a fundamental challenge in unsupervised goal-conditioned reinforcement learning. While selecting exploratory goals at the frontier of previously explored states is an effective strategy, the policy during training may still have limited capability of reaching rare goals on the frontier, resulting in reduced exploratory behavior. We propose "Cluster Edge Exploration" (CE2), a new goal-directed exploration algorithm that when choosing goals in sparsely explored areas of the state space gives priority to goal states that remain accessible to the agent. The key idea is clustering to group states that are easily reachable from one another by the current policy under training in a latent space and traversing to states holding significant exploration potential on the boundary of these clusters before doing exploratory behavior. In challenging robotics environments including navigating a maze with a multi-legged ant robot, manipulating objects with a robot arm on a cluttered tabletop, and rotating objects in the palm of an anthropomorphic robotic hand, CE2 demonstrates superior efficiency in exploration compared to baseline methods and ablations.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 41 of its 48 references that are in the map. First posted on arXiv 3 Nov 2024.
Builds on
41 of 48 references.
References · 41
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2017Hindsight Experience Replaycited by 1,096
- 2020Mastering Atari with Discrete World Modelscited by 990
- 2018Exploration by Random Network Distillationcited by 966
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
- 2020Planning to Explore via Self-Supervised World Modelscited by 365
29 more
- 2022Transformers are Sample-Efficient World Modelscited by 317
- 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
- 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
- 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
- 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
- 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
- 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
- 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
- 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
- 2021Discovering and Achieving Goals via World Modelscited by 131
- 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
- 2018Model-Based Active Explorationcited by 105
- 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
- 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 88
- 2020Automatic Curriculum Learning through Value Disagreementcited by 86
- 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
- 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
- 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
- 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
- 2019Exploration via Hindsight Goal Generationcited by 53
- 2023Planning Goals for Explorationcited by 28
- 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
- 2021When should agents explore?cited by 20
- 2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19
- 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 17
- 2023Curious Replay for Model-based Adaptationcited by 16
- 2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.