Goal-Conditioned Supervised Learning with Sub-Goal Prediction
Not on the map. Why?
Abstract (arXiv)
Recently, a simple yet effective algorithm -- goal-conditioned supervised-learning (GCSL) -- was proposed to tackle goal-conditioned reinforcement-learning. GCSL is based on the principle of hindsight learning: by observing states visited in previously executed trajectories and treating them as attained goals, GCSL learns the corresponding actions via supervised learning. However, GCSL only learns a goal-conditioned policy, discarding other information in the process. Our insight is that the same hindsight principle can be used to learn to predict goal-conditioned sub-goals from the same trajectory. Based on this idea, we propose Trajectory Iterative Learner (TraIL), an extension of GCSL that further exploits the information in a trajectory, and uses it for learning to predict both actions and sub-goals. We investigate the settings in which TraIL can make better use of the data, and discover that for several popular problem settings, replacing real goals in GCSL with predicted TraIL sub-goals allows the agent to reach a greater set of goal states using the exact same data as GCSL, thereby improving its overall performance.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 17 May 2023.
Builds on
19 of 32 references.
References · 19
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2017Hindsight Experience Replaycited by 1,136
- 2017Deep Reinforcement Learning that Matterscited by 754
- 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
- 2018Visual Reinforcement Learning with Imagined Goalscited by 251
- 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
- 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
- 2018Motion Planning Networkscited by 80
7 more
- 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 62
- 2017Hindsight policy gradientscited by 37
- 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 29
- 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
- 2019Harnessing Reinforcement Learning for Neural Motion Planningcited by 21
- 2020PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goalscited by 21
- 2020Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planningcited by 19
Built on by
Citing papers per year: 2024: 1.
1 paper.