mlsanityData to 7 Oct 2026

Average-reward model-free reinforcement learning: a systematic review and literature mapping

arXiv 2010.08920 ↗First posted 18 Oct 2020cs.LG cs.AIcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning is important part of artificial intelligence. In this paper, we review model-free reinforcement learning that utilizes the average reward optimality criterion in the infinite horizon setting. Motivated by the solo survey by Mahadevan (1996a), we provide an updated review of work in this area and extend it to cover policy-iteration and function approximation methods (in addition to the value-iteration and tabular counterparts). We present a comprehensive literature mapping. We also identify and discuss opportunities for future work.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 138 references that are in the map. First posted on arXiv 18 Oct 2020.

Builds on

36 of 138 references.

References · 36

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  9. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  10. 2017A unified view of entropy-regularized Markov decision processescited by 174
  11. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  12. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
24 more
  1. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  2. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  3. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  4. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 119
  5. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
  6. 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
  7. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  8. 2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 53
  9. 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
  10. 2020Whittle index based Q-learning for restless bandits with average rewardcited by 43
  11. 2017Primal-Dual π Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problemscited by 32
  12. 2019Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspectivecited by 26
  13. 2016Combining policy gradient and Q-learningcited by 23
  14. 2013PEGASUS: A Policy Search Method for Large MDPs and POMDPscited by 21
  15. 2019Deep Residual Reinforcement Learningcited by 21
  16. 2019Momentum in Reinforcement Learningcited by 19
  17. 2018Model-Free Linear Quadratic Control via Reduction to Expert Predictioncited by 10
  18. 2020Adaptive Approximate Policy Iterationcited by 9
  19. 2017Investigating Recurrence and Eligibility Traces in Deep Q-Networkscited by 8
  20. 2020A Model-free Learning Algorithm for Infinite-horizon Average-reward MDPs with Near-optimal Regretcited by 7
  21. 2020Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applicationscited by 6
  22. 2016A Batch, Off-Policy, Actor-Critic Algorithm for Optimizing the Average Rewardcited by 5
  23. 2018Differential Temporal Difference Learningcited by 2
  24. 2016Differential TD Learning for Value Function Approximationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 1; 2023: 4; 2024: 5; 2025: 5; 2026: 1.

19 papers.

Sort:

By area:

  1. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  2. 2023Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factorcited by 13
  3. 2024Reinforcement Learning with LTL and ω-Regular Objectives via Optimality-Preserving Translation to Average Rewardscited by 13
  4. 2023AutoRL Hyperparameter Landscapescited by 8
  5. 2021Examining average and discounted reward optimality criteria in reinforcement learningcited by 6
  6. 2023Full Gradient Deep Reinforcement Learning for Average-Reward Criterioncited by 5
  7. 2024Provable Policy Gradient Methods for Average-Reward Markov Potential Gamescited by 5
  8. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 3
  9. 2021Multi-agent Natural Actor-critic Reinforcement Learning Algorithmscited by 2
  10. 2022Slowly Changing Adversarial Bandit Algorithms are Efficient for Discounted MDPscited by 2
  11. 2024Burning RED: Unlocking Subtask-Driven Reinforcement Learning and Risk-Awareness in Average-Reward Markov Decision Processescited by 2
  12. 2025Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysiscited by 2
7 more
  1. 2025Implicit Updates for Average-Reward Temporal Difference Learningcited by 2
  2. 2023Racing Towards Reinforcement Learning based control of an Autonomous Formula SAE Carcited by 1
  3. 2025Efficient Computation of Blackwell Optimal Policies using Rational Functionscited by 1
  4. 2025Finite-Time Bounds for Average-Reward Fitted Q-Iterationcited by 1
  5. 2024Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Taskscited by 0
  6. 2025Average Reward Reinforcement Learning for Omega-Regular and Mean-Payoff Objectivescited by 0
  7. 2026A Harmonic Mean Formulation of Average Reward Reinforcement Learning in SMDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Inverse Reinforcement Learning with the Average Reward Criterion
  2. 2020On Reward-Free Reinforcement Learning with Linear Function Approximation
  3. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  4. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
  5. 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
  6. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
  7. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  8. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models