mlsanityData to 14 Oct 2026

Goal-Conditioned Agents that Learn Everything All at Once

arXiv 2605.23551 ↗First posted 22 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

A goal-conditioned reinforcement learning agent exploring an environment will see a wealth of information throughout a trajectory, most of which is discarded when only performing on-policy updates with respect to the commanded goal. All-goals learning, where each transition is used for learning off-policy with respect to every goal, allows agents to extract maximal information, however it is usually computationally infeasible when done via naive relabelling. This can be overcome by jointly outputting values and actions for every goal at once, allowing for efficient, parallel all-goals updates with a single pass through the network, in a process we call Learning Everything all at Once (LEO). We show that this approach significantly outperforms other methods on goal-conditioned Craftax and is competitive with existing baselines on continuous control environments, while achieving a >250x speed-up compared to all-goals relabelling. We then go on to show that this approach can be made even more powerful by using LEO as a teacher network, rather than a direct actor. We hope that, by unlocking all-goals learning at scale, LEO can serve as a useful tool for RL practitioners in complex environments. We open source our code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 64 of its 82 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

64 of 82 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Scalable Option Learning in High-Throughput Environments30 Aug
  2. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning16 Jun
  3. 2025Horizon Reduction Makes RL Scalable4 Jun
  4. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models15 Apr
  5. 2024Accelerating Goal-Conditioned RL Algorithms and Research20 Aug
  6. 2024Simplifying Deep Temporal Difference Learning5 Jul

Other references · 58

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  12. 2017Hindsight Experience Replaycited by 1,136
38 more
  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  4. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  5. 2016The Option-Critic Architecturecited by 448
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  8. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  10. 2019Learning Latent Plans from Playcited by 316
  11. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  12. 2020First return, then explorecited by 266
  13. 2019Self-Supervised Exploration via Disagreementcited by 265
  14. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  15. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  16. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  17. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  18. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  19. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  20. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  21. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  22. 2016Learning to Navigate in Complex Environmentscited by 198
  23. 2020The NetHack Learning Environmentcited by 198
  24. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 191
  25. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  26. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  27. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  28. 2021Learning One Representation to Optimize All Rewardscited by 115
  29. 2018Universal Successor Features Approximatorscited by 103
  30. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  31. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  32. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  33. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  34. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  35. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 85
  36. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  37. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  38. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 60

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Making Universal Policies Universal
  2. 2025HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents
  3. 2025Environment Agnostic Goal-Conditioning, A Study of Reward-Free Autonomous Learning
  4. 2019Learning to Reach Goals via Iterated Supervised Learning
  5. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  6. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  7. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning
  8. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space