mlsanityData to 7 Oct 2026

Average-Reward Soft Actor-Critic

arXiv 2501.09080 ↗First posted 15 Jan 2025cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

The average-reward formulation of reinforcement learning (RL) has drawn increased interest in recent years for its ability to solve temporally-extended problems without relying on discounting. Meanwhile, in the discounted setting, algorithms with entropy regularization have been developed, leading to improvements over deterministic methods. Despite the distinct benefits of these approaches, deep RL algorithms for the entropy-regularized average-reward objective have not been developed. While policy-gradient based approaches have recently been presented for the average-reward literature, the corresponding actor-critic framework remains less explored. In this paper, we introduce an average-reward soft actor-critic algorithm to address these gaps in the field. We validate our method by comparing with existing average-reward algorithms on standard RL benchmarks, achieving superior performance for the average-reward criterion.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

38 of 63 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
26 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  4. 2019A Theory of Regularized Markov Decision Processescited by 279
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  6. 2017A unified view of entropy-regularized Markov decision processescited by 174
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  8. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  9. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  10. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  11. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  12. 2020Learning Infinite-horizon Average-reward MDPs with Linear Function Approximationcited by 48
  13. 2022Stochastic first-order methods for average-reward Markov decision processescited by 30
  14. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  15. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  16. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 25
  17. 2020A reinforcement learning approach to rare trajectory samplingcited by 21
  18. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  19. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  20. 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10
  21. 2022Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learningcited by 8
  22. 2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
  23. 2022Making Reinforcement Learning Work on Swimmercited by 6
  24. 2021Entropy Regularized Reinforcement Learning Using Large Deviation Theorycited by 5
  25. 2024RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learningcited by 5
  26. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  2. 2025Revisiting Value Iteration: Unified Analysis of Discounted and Average-Reward Casescited by 1
  3. 2026Maximum Entropy Exploration Without the Rolloutscited by 1
  4. 2026Active Inference as a Convex Markov Decision Processcited by 1
  5. 2026Thermodynamics of Reinforcement Learning Curriculacited by 0
  6. 2026Deep Reinforcement Learning and The Tale of Two Temporal Difference Errorscited by 0