mlsanityData to 14 Oct 2026

Effective Reinforcement Learning Control using Conservative Soft Actor-Critic

arXiv 2505.03356 ↗First posted 6 May 2025cs.ROcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement Learning (RL) has shown great potential in complex control tasks, particularly when combined with deep neural networks within the Actor-Critic (AC) framework. However, in practical applications, balancing exploration, learning stability, and sample efficiency remains a significant challenge. Traditional methods such as Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) address these issues by incorporating entropy or relative entropy regularization, but often face problems of instability and low sample efficiency. In this paper, we propose the Conservative Soft Actor-Critic (CSAC) algorithm, which seamlessly integrates entropy and relative entropy regularization within the AC framework. CSAC improves exploration through entropy regularization while avoiding overly aggressive policy updates with the use of relative entropy regularization. Evaluations on benchmark tasks and real-world robotic simulations demonstrate that CSAC offers significant improvements in stability and efficiency over existing methods. These findings suggest that CSAC provides strong robustness and application potential in control tasks under dynamic environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 48 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

20 of 48 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 142
  9. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  10. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 59
  11. 2020Softmax Deep Double Deterministic Policy Gradientscited by 32
  12. 2019Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy Optimizationcited by 17
8 more
  1. 2020Learning Object-conditioned Exploration using Distributed Soft Actor Criticcited by 12
  2. 2023PyFlyt -- UAV Simulation Environments for Reinforcement Learning Researchcited by 9
  3. 2024Arm-Constrained Curriculum Learning for Loco-Manipulation of the Wheel-Legged Robotcited by 8
  4. 2021Hybrid Car-Following Strategy based on Deep Deterministic Policy Gradient and Cooperative Adaptive Cruise Controlcited by 6
  5. 2021Multi-agent Soft Actor-Critic Based Hybrid Motion Planner for Mobile Robotscited by 5
  6. 2023Effective Multi-Agent Deep Reinforcement Learning Control with Relative Entropy Regularizationcited by 5
  7. 2023SACPlanner: Real-World Collision Avoidance with a Soft Actor Critic Local Planner and Polar State Representationscited by 4
  8. 2023Risk-anticipatory autonomous driving strategies considering vehicles' weights, based on hierarchical deep reinforcement learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Average-Reward Soft Actor-Critic
  2. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  3. 2026FERPO: Forward Entropy-Regularized Policy Optimization
  4. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
  5. 2025SACn: Soft Actor-Critic with n-step Returns
  6. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
  7. 2026Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning
  8. 2022Latent State Marginalization as a Low-cost Approach for Improving Exploration