mlsanityData to 14 Oct 2026

Convergence and stability of Q-learning in Hierarchical Reinforcement Learning

arXiv 2511.17351 ↗First posted 21 Nov 2025cs.LG cs.SY eess.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Hierarchical Reinforcement Learning promises, among other benefits, to efficiently capture and utilize the temporal structure of a decision-making problem and to enhance continual learning capabilities, but theoretical guarantees lag behind practice. In this paper, we propose a Feudal Q-learning scheme and investigate under which conditions its coupled updates converge and are stable. By leveraging the theory of Stochastic Approximation and the ODE method, we present a theorem stating the convergence and stability properties of Feudal Q-learning. This provides a principled convergence and stability analysis tailored to Feudal RL. Moreover, we show that the updates converge to a point that can be interpreted as an equilibrium of a suitably defined game, opening the door to game-theoretic approaches to Hierarchical RL. Lastly, experiments based on the Feudal Q-learning algorithm support the outcomes anticipated by theory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 43 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

15 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximation27 Apr

Other references · 14

  1. 2018Is Q-learning Provably Efficient?cited by 487
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2016The Option-Critic Architecturecited by 448
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  6. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 142
  7. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 94
  8. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  9. 2020Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performancecited by 35
  10. 2018Learning Abstract Optionscited by 31
  11. 2019Diversity-enabled sweet spots in layered architectures and speed-accuracy trade-offs in sensorimotor controlcited by 7
  12. 2024Towards a Systems Theory of Algorithmscited by 7
2 more
  1. 2021Multi Timescale Stochastic Approximation: Stability and Convergencecited by 5
  2. 2022Safe Hierarchical Model Predictive Control and Planning for Autonomous Systemscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Formalizing Reinforcement Learning Theory: A Robbins-Siegmund Approach
  2. 2024On the Stability of Learning in Network Games with Many Players
  3. 2026Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
  4. 2024Coarse Q-learning: Indifference, Indeterminacy, and Instability
  5. 2025Convergent Q-Learning for Infinite-Horizon General-Sum Markov Games through Behavioral Economics
  6. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
  7. 2024Mutation-Bias Learning in Games
  8. 2025Is Q-learning an Ill-posed Problem?