mlsanityData to 7 Oct 2026

Average-Reward Learning and Planning with Options

arXiv 2110.13855 ↗First posted 26 Oct 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We extend the options framework for temporal abstraction in reinforcement learning from discounted Markov decision processes (MDPs) to average-reward MDPs. Our contributions include general convergent off-policy inter-option learning algorithms, intra-option algorithms for learning values and models, as well as sample-based planning variants of our learning algorithms. Our algorithms and convergence proofs extend those recently developed by Wan, Naik, and Sutton. We also extend the notion of option-interrupting behavior from the discounted to the average-reward formulation. We show the efficacy of the proposed algorithms with experiments on a continuing version of the Four-Room domain.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 9 of its 34 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

9 of 34 references.

References · 9

  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  2. 2016The Option-Critic Architecturecited by 433
  3. 2016Variational Intrinsic Controlcited by 262
  4. 2018Variational Option Discovery Algorithmscited by 121
  5. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  6. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  7. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  8. 2021Discovery of Options via Meta-Learned Subgoalscited by 27
  9. 2017Exploration--Exploitation in MDPs with Optionscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 4; 2025: 1; 2026: 1.

8 papers.

  1. 2022Goal-Space Planning with Subgoal Modelscited by 12
  2. 2024Asymptotic and Finite Sample Analysis of Nonexpansive Stochastic Approximations with Markovian Noisecited by 10
  3. 2022On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPscited by 5
  4. 2024Asynchronous Stochastic Approximation with Applications to Average-Reward Reinforcement Learningcited by 2
  5. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processescited by 2
  6. 2024Hierarchical Average-Reward Linearly-solvable Markov Decision Processescited by 1
  7. 2025Average-reward reinforcement learning in semi-Markov decision processes via relative value iterationcited by 1
  8. 2024A New View on Planning in Online Reinforcement Learningcited by 0