mlsanityData to 14 Oct 2026

Hierarchical Average-Reward Linearly-solvable Markov Decision Processes

arXiv 2407.06690 ↗First posted 9 Jul 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We introduce a novel approach to hierarchical reinforcement learning for Linearly-solvable Markov Decision Processes (LMDPs) in the infinite-horizon average-reward setting. Unlike previous work, our approach allows learning low-level and high-level tasks simultaneously, without imposing limiting restrictions on the low-level tasks. Our method relies on partitions of the state space that create smaller subtasks that are easier to solve, and the equivalence between such partitions to learn more efficiently. We then exploit the compositionality of low-level tasks to exactly represent the value function of the high-level task. Experiments show that our approach can outperform flat average-reward reinforcement learning by one or several orders of magnitude.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 27 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

9 of 27 references.

References · 9

  1. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  4. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
  5. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  6. 2017Exploration--Exploitation in MDPs with Optionscited by 19
  7. 2021Average-Reward Learning and Planning with Optionscited by 8
  8. 2021Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processescited by 6
  9. 2016Hierarchical Linearly-Solvable Markov Decision Problemscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Terminal Representation in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  2. 2026Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach
  3. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
  4. 2024Reinforcement Learning with Options and State Representation
  5. 2024Reinforcement Learning for Infinite-Horizon Average-Reward Linear MDPs via Approximation by Discounted-Reward MDPs
  6. 2026A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPs
  7. 2024Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processes
  8. 2024On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes