# Many-Goals Reinforcement Learning

arXiv: https://arxiv.org/abs/1806.09605 · HTML: https://mlsanity.com/paper/1806.09605/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2018.

## Built on by (most cited first, 22 of 22)

- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data](https://mlsanity.com/paper/2210.10047/) (2022)
- [Return-Based Contrastive Representation Learning for Reinforcement Learning](https://mlsanity.com/paper/2102.10960/) (2021)
- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [Pre-Training and Fine-Tuning Generative Flow Networks](https://mlsanity.com/paper/2310.03419/) (2023)
- [Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.12030/) (2022)
- [Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results](https://mlsanity.com/paper/2212.06759/) (2022)
- [Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning](https://mlsanity.com/paper/2211.00247/) (2022)
- [Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets](https://mlsanity.com/paper/2406.01150/) (2024)
- [Open-Ended Reinforcement Learning with Neural Reward Functions](https://mlsanity.com/paper/2202.08266/) (2022)
- [Reset-free Reinforcement Learning with World Models](https://mlsanity.com/paper/2408.09807/) (2024)
- [Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics](https://mlsanity.com/paper/2111.07908/) (2021)
- [World Value Functions: Knowledge Representation for Multitask Reinforcement Learning](https://mlsanity.com/paper/2205.08827/) (2022)
- [From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agent](https://mlsanity.com/paper/2208.04833/) (2022)
- [Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance](https://mlsanity.com/paper/2409.03996/) (2024)
- [Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines](https://mlsanity.com/paper/2507.05561/) (2025)
- [Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization](https://mlsanity.com/paper/2605.06145/) (2026)
- [Learning user-defined sub-goals using memory editing in reinforcement learning](https://mlsanity.com/paper/2205.00399/) (2022)
- [World Value Functions: Knowledge Representation for Learning and Planning](https://mlsanity.com/paper/2206.11940/) (2022)
- [Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery](https://mlsanity.com/paper/2609.36473/) (2026)
- [Goal-Conditioned Agents that Learn Everything All at Once](https://mlsanity.com/paper/2605.23551/) (2026)

## Abstract (arXiv)

All-goals updating exploits the off-policy nature of Q-learning to update all possible goals an agent could have from each transition in the world, and was introduced into Reinforcement Learning (RL) by Kaelbling (1993). In prior work this was mostly explored in small-state RL problems that allowed tabular representations and where all possible goals could be explicitly enumerated and learned separately. In this paper we empirically explore 3 different extensions of the idea of updating many (instead of all) goals in the context of RL with deep neural networks (or DeepRL for short). First, in a direct adaptation of Kaelbling's approach we explore if many-goals updating can be used to achieve mastery in non-tabular visual-observation domains. Second, we explore whether many-goals updating can be used to pre-train a network to subsequently learn faster and better on a single main task of interest. Third, we explore whether many-goals updating can be used to provide auxiliary task updates in training a network to learn faster and better on a single main task of interest. We provide comparisons to baselines for each of the 3 extensions.