mlsanityData to 7 Oct 2026

RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning

arXiv 2408.01972 ↗First posted 4 Aug 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose an off-policy deep reinforcement learning (DRL) method utilizing the average reward criterion. While most existing DRL methods employ the discounted reward criterion, this can potentially lead to a discrepancy between the training objective and performance metrics in continuing tasks, making the average reward criterion a recommended alternative. We introduce RVI-SAC, an extension of the state-of-the-art off-policy DRL method, Soft Actor-Critic (SAC), to the average reward criterion. Our proposal consists of (1) Critic updates based on RVI Q-learning, (2) Actor updates introduced by the average reward soft policy improvement theorem, and (3) automatic adjustment of Reset Cost enabling the average reward reinforcement learning to be applied to tasks with termination. We apply our method to the Gymnasium's Mujoco tasks, a subset of locomotion tasks, and demonstrate that RVI-SAC shows competitive performance compared to existing methods.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 4 Aug 2024.

Builds on

18 of 35 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  10. 2018Maximum a Posteriori Policy Optimisationcited by 318
  11. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  12. 2021Breaking the Deadly Triad with a Target Networkcited by 47
6 more
  1. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  2. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  3. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 25
  4. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  5. 2021Examining average and discounted reward optimality criteria in reinforcement learningcited by 6
  6. 2022On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Average-Reward Soft Actor-Criticcited by 6
  2. 2025An Empirical Study of Deep Reinforcement Learning in Continuing Taskscited by 1
  3. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  4. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  5. 2026Thermodynamics of Reinforcement Learning Curriculacited by 0