mlsanityData to 7 Oct 2026

Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience

arXiv 2109.11767 ↗First posted 24 Sep 2021cs.LG cs.SY eess.SYcited by 8

Not on the map. Why?

Abstract (arXiv)

Soft Actor-Critic (SAC) is an off-policy actor-critic reinforcement learning algorithm, essentially based on entropy regularization. SAC trains a policy by maximizing the trade-off between expected return and entropy (randomness in the policy). It has achieved state-of-the-art performance on a range of continuous-control benchmark tasks, outperforming prior on-policy and off-policy methods. SAC works in an off-policy fashion where data are sampled uniformly from past experiences (stored in a buffer) using which parameters of the policy and value function networks are updated. We propose certain crucial modifications for boosting the performance of SAC and make it more sample efficient. In our proposed improved SAC, we firstly introduce a new prioritization scheme for selecting better samples from the experience replay buffer. Secondly we use a mixture of the prioritized off-policy data with the latest on-policy data for training the policy and the value function networks. We compare our approach with the vanilla SAC and some recent variants of SAC and show that our approach outperforms the said algorithmic benchmarks. It is comparatively more stable and sample efficient when tested on a number of continuous control tasks in MuJoCo environments.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

38 of 57 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016OpenAI Gymcited by 1,684
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2015Prioritized Experience Replaycited by 1,070
26 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  7. 2018Distributed Prioritized Experience Replaycited by 260
  8. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  9. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  10. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  11. 2018Remember and Forget for Experience Replaycited by 42
  12. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 42
  13. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  14. 2018Episodic Memory Deep Q-Networkscited by 41
  15. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  16. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
  17. 2018Feedback Control For Cassie With Deep Reinforcement Learningcited by 36
  18. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  19. 2018Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systemscited by 21
  20. 2019Prioritized Sequence Experience Replaycited by 20
  21. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19
  22. 2019Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcited by 16
  23. 2019Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environmentscited by 13
  24. 2018On Passivity, Reinforcement Learning and Higher-Order Learning in Multi-Agent Finite Gamescited by 8
  25. 2018ViZDoom: DRQN with Prioritized Experience Replay, Double-Q Learning, & Snapshot Ensemblingcited by 3
  26. 2020Band-limited Soft Actor Critic Modelcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 3; 2025: 2.

8 papers.

  1. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 22
  2. 2022Revisiting Discrete Soft Actor-Criticcited by 11
  3. 2022Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel Statescited by 3
  4. 2024Revisiting Experience Replayable Conditionscited by 3
  5. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
  6. 2024Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Trainingcited by 1
  7. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learningcited by 1
  8. 2025Mining--Gym: A Configurable RL Benchmarking Environment for Truck Dispatch Schedulingcited by 0