mlsanityData to 14 Oct 2026

Q-Shaped Options for Hierarchical Reinforcement Learning

arXiv 2610.12135 ↗First posted 8 Oct 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations also fits: Interpretable and logic-guided RL

Abstract (arXiv)

Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states. In principle, Hierarchical Reinforcement Learning (HRL) addresses both challenges through the interaction between action (temporal) and state (spatial) abstraction. First, using an action abstraction to represent temporally extended behaviour as options reduces the effective decision horizon. Second, enabling different state abstractions at each level of the decision process permits greater data aggregation for learning. However, realising these two benefits of a hierarchical policy depends on learning an appropriate action abstraction. Current HRL algorithms fail in one of two ways. Some discard distinctions between options needed for optimal control, undermining hierarchy altogether. Others retain unnecessary distinctions, preserving horizon reduction, but forfeiting coarser state abstraction. In this work, we characterise three desiderata for an action abstraction. We introduce Q-Shaped Options (QSO) to address all three. QSO builds on an architecture with distinct state-value functions, Q functions and policies at each level of the hierarchy. It learns the action abstraction between consecutive levels as a shared encoder shaped by their respective Q functions. The low-level Q function uses the option as a goal, encouraging the abstraction to retain distinctions necessary for optimal control. The high-level Q function uses it as an action, encouraging unnecessary distinctions to be discarded. Across offline goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance in tasks where all other evaluated algorithms fail.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 41 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

20 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Reinforcement Learning with Action Chunking10 Jul
  2. 2025Horizon Reduction Makes RL Scalable4 Jun

Other references · 18

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2016The Option-Critic Architecturecited by 448
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
6 more
  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  3. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  4. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  6. 2017Near Optimal Behavior via Approximate State Abstractioncited by 66

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  2. 2024Reinforcement Learning with Options and State Representation
  3. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  4. 2024Reconciling Spatial and Temporal Abstractions for Goal Representation
  5. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
  6. 2024Temporal Abstraction in Reinforcement Learning with Offline Data
  7. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Control
  8. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions