mlsanityData to 14 Oct 2026

Scalable Option Learning in High-Throughput Environments

arXiv 2509.00338 ↗First posted 30 Aug 2025cs.LG cs.AIcited by 5

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Hierarchical reinforcement learning (RL) has the potential to enable effective decision-making over long timescales. Existing approaches, while promising, have yet to realize the benefits of large-scale training. In this work, we identify and solve several key challenges in scaling online hierarchical RL to high-throughput environments. We propose Scalable Option Learning (SOL), a highly scalable hierarchical RL algorithm which achieves a ~35x higher throughput compared to existing hierarchical methods. To demonstrate SOL's performance and scalability, we train hierarchical agents using 30 billion frames of experience on the complex game of NetHack, significantly surpassing flat agents and demonstrating positive scaling trends. We also validate SOL on MiniHack and Mujoco environments, showcasing its general applicability. Our code is open sourced at: github.com/facebookresearch/sol.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 47 of its 78 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

47 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning16 Jun

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
34 more
  1. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  3. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  4. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  6. 2016The Option-Critic Architecturecited by 448
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  8. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  9. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  10. 2023Reward Design with Language Modelscited by 261
  11. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  12. 2020The NetHack Learning Environmentcited by 198
  13. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  14. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  15. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  16. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  17. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  18. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  19. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  20. 2022Exploration via Elliptical Episodic Bonusescited by 51
  21. 2023Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulationcited by 49
  22. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  23. 2023ASC: Adaptive Skill Coordination for Robotic Mobile Manipulationcited by 42
  24. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  25. 2018Learning Abstract Optionscited by 31
  26. 2021Flexible Option Learningcited by 25
  27. 2017Learnings Options End-to-End for Continuous Action Taskscited by 24
  28. 2024PufferLib: Making Reinforcement Learning Libraries and Environments Play Nicecited by 24
  29. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  30. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 21
  31. 2017Learning Multi-Level Hierarchies with Hindsightcited by 20
  32. 2020RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problemcited by 16
  33. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  34. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Groundingcited by 5
  2. 2025SLAP: Shortcut Learning for Abstract Planningcited by 3
  3. 2025MIMIC-MJX: Neuromechanical Emulation of Animal Behaviorcited by 2
  4. 2026Hierarchical Behaviour Spacescited by 0
  5. 2026Goal-Conditioned Agents that Learn Everything All at Oncecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning
  2. 2024Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration ↗
  3. 2024Reinforcement Learning with Options and State Representation
  4. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
  5. 2025Horizon Reduction Makes RL Scalable
  6. 2026Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
  7. 2024Hierarchical Transformers are Efficient Meta-Reinforcement Learners
  8. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information