mlsanityData to 14 Oct 2026

Hierarchical Behaviour Spaces

arXiv 2604.24558 ↗First posted 27 Apr 2026cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recent work in hierarchical reinforcement learning has shown success in scaling to billions of timesteps when learning over a set of predefined option reward functions. We show that, instead of using a single reward function per option, the reward functions can be effectively used to induce a space of behaviours, by letting the controller specify linear combinations over reward functions, allowing a more expressive set of policies to be represented. We call this method Hierarchical Behaviour Spaces (HBS). We evaluate HBS on the NetHack Learning Environment, demonstrating strong performance. We conduct a series of experiments and determine that, perhaps going against conventional wisdom, the benefits of hierarchy in our method come from increased exploration rather than long term reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 37 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

29 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Scalable Option Learning in High-Throughput Environments30 Aug
  2. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning16 Jun
  3. 2025Horizon Reduction Makes RL Scalable4 Jun
  4. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  3. 2016The Option-Critic Architecturecited by 448
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  5. 2016Variational Intrinsic Controlcited by 267
  6. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  7. 2020The NetHack Learning Environmentcited by 198
  8. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  9. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  10. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  11. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  12. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
13 more
  1. 2022Exploration via Elliptical Episodic Bonusescited by 51
  2. 2022Skill-based Model-based Reinforcement Learningcited by 50
  3. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  4. 2021Flexible Option Learningcited by 25
  5. 2017Learnings Options End-to-End for Continuous Action Taskscited by 24
  6. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  7. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  8. 2022Insights From the NeurIPS 2021 NetHack Challengecited by 20
  9. 2017Exploration--Exploitation in MDPs with Optionscited by 19
  10. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  11. 2023NetHack is Hard to Hackcited by 11
  12. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  13. 2022Hierarchical Kickstarting for Skill Transfer in Reinforcement Learningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning with Options and State Representation
  2. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  3. 2026Q-Shaped Options for Hierarchical Reinforcement Learning
  4. 2023Deep Reinforcement Learning from Hierarchical Preference Design
  5. 2024Hierarchical Average-Reward Linearly-solvable Markov Decision Processes
  6. 2018Hierarchical Imitation and Reinforcement Learning
  7. 2024On the benefits of pixel-based hierarchical policies for task generalization
  8. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventions