# Average-Reward Learning and Planning with Options

arXiv: https://arxiv.org/abs/2110.13855 · HTML: https://mlsanity.com/paper/2110.13855/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [A Laplacian Framework for Option Discovery in Reinforcement Learning](https://mlsanity.com/paper/1703.00956/) (2017)
- [Learning and Planning in Average-Reward Markov Decision Processes](https://mlsanity.com/paper/2006.16318/) (2020)
- [Average-Reward Off-Policy Policy Evaluation with Function Approximation](https://mlsanity.com/paper/2101.02808/) (2021)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [Exploration--Exploitation in MDPs with Options](https://mlsanity.com/paper/1703.08667/) (2017)

## Built on by (most cited first, 8 of 8)

- [Goal-Space Planning with Subgoal Models](https://mlsanity.com/paper/2206.02902/) (2022)
- [Asymptotic and Finite Sample Analysis of Nonexpansive Stochastic Approximations with Markovian Noise](https://mlsanity.com/paper/2409.19546/) (2024)
- [On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPs](https://mlsanity.com/paper/2209.15141/) (2022)
- [Asynchronous Stochastic Approximation with Applications to Average-Reward Reinforcement Learning](https://mlsanity.com/paper/2409.03915/) (2024)
- [Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes](https://mlsanity.com/paper/2602.16629/) (2026)
- [Hierarchical Average-Reward Linearly-solvable Markov Decision Processes](https://mlsanity.com/paper/2407.06690/) (2024)
- [Average-reward reinforcement learning in semi-Markov decision processes via relative value iteration](https://mlsanity.com/paper/2512.06218/) (2025)
- [A New View on Planning in Online Reinforcement Learning](https://mlsanity.com/paper/2406.01562/) (2024)

## Abstract (arXiv)

We extend the options framework for temporal abstraction in reinforcement learning from discounted Markov decision processes (MDPs) to average-reward MDPs. Our contributions include general convergent off-policy inter-option learning algorithms, intra-option algorithms for learning values and models, as well as sample-based planning variants of our learning algorithms. Our algorithms and convergence proofs extend those recently developed by Wan, Naik, and Sutton. We also extend the notion of option-interrupting behavior from the discounted to the average-reward formulation. We show the efficacy of the proposed algorithms with experiments on a continuing version of the Four-Room domain.