mlsanityData to 14 Oct 2026

On-Policy Deep Reinforcement Learning for the Average-Reward Criterion

arXiv 2106.07329 ↗First posted 14 Jun 2021cs.LG cs.AI stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

We develop theory and algorithms for average-reward on-policy Reinforcement Learning (RL). We first consider bounding the difference of the long-term average reward for two policies. We show that previous work based on the discounted return (Schulman et al., 2015; Achiam et al., 2017) results in a non-meaningful bound in the average-reward setting. By addressing the average-reward criterion directly, we then derive a novel bound which depends on the average divergence between the two policies and Kemeny's constant. Based on this bound, we develop an iterative procedure which produces a sequence of monotonically improved policies for the average reward criterion. This iterative procedure can then be combined with classic DRL (Deep Reinforcement Learning) methods, resulting in practical DRL algorithms that target the long-run average reward criterion. In particular, we demonstrate that Average-Reward TRPO (ATRPO), which adapts the on-policy TRPO algorithm to the average-reward criterion, significantly outperforms TRPO in the most challenging MuJuCo environments.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 20 of its 65 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

20 of 65 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2021Convex optimizationcited by 2,515
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2017Constrained Policy Optimizationcited by 983
  10. 2018Reward Constrained Policy Optimizationcited by 417
  11. 2020Projection-Based Constrained Policy Optimizationcited by 203
  12. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
8 more
  1. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  2. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  3. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  4. 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 82
  5. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  6. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 45
  7. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 26
  8. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 4; 2024: 5; 2025: 8; 2026: 4.

30 papers.

Sort:

By area:

  1. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  2. 2022Stochastic first-order methods for average-reward Markov decision processescited by 32
  3. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  4. 2023Robust Average-Reward Markov Decision Processescited by 22
  5. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  6. 2024Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oraclescited by 6
  7. 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 6
  8. 2025Average-Reward Soft Actor-Criticcited by 6
  9. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 5
  10. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 4
  11. 2021Refined Policy Improvement Bounds for MDPscited by 3
  12. 2022Adaptive patch foraging in deep reinforcement learning agentscited by 3
18 more
  1. 2026Policy Gradient Algorithms in Average-Reward Multichain MDPscited by 3
  2. 2022Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Spacecited by 2
  3. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimalitycited by 2
  4. 2025Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learningcited by 2
  5. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processescited by 2
  6. 2022Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problemscited by 1
  7. 2022Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learningcited by 1
  8. 2022Deriving time-averaged active inference from control principlescited by 1
  9. 2024Intervention-Assisted Policy Gradient Methods for Online Stochastic Queuing Network Optimization: Technical Reportcited by 1
  10. 2025An Empirical Study of Deep Reinforcement Learning in Continuing Taskscited by 1
  11. 2025Follow the STARs: Dynamic ω-Regular Shielding of Learned Policiescited by 1
  12. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  13. 2026Ergodicity in reinforcement learningcited by 1
  14. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  15. 2022Processing Network Controls via Deep Reinforcement Learningcited by 0
  16. 2022Improving Sample Efficiency in Evolutionary RL Using Off-Policy Rankingcited by 0
  17. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  18. 2025Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Gamescited by 0