# Effective Reinforcement Learning Control using Conservative Soft Actor-Critic

arXiv: https://arxiv.org/abs/2505.03356 · HTML: https://mlsanity.com/paper/2505.03356/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [panda-gym: Open-source goal-conditioned environments for robotic learning](https://mlsanity.com/paper/2106.13687/) (2021)
- [Softmax Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2010.09177/) (2020)
- [Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy Optimization](https://mlsanity.com/paper/1911.05478/) (2019)
- [Learning Object-conditioned Exploration using Distributed Soft Actor Critic](https://mlsanity.com/paper/2007.14545/) (2020)
- [PyFlyt -- UAV Simulation Environments for Reinforcement Learning Research](https://mlsanity.com/paper/2304.01305/) (2023)
- [Arm-Constrained Curriculum Learning for Loco-Manipulation of the Wheel-Legged Robot](https://mlsanity.com/paper/2403.16535/) (2024)
- [Hybrid Car-Following Strategy based on Deep Deterministic Policy Gradient and Cooperative Adaptive Cruise Control](https://mlsanity.com/paper/2103.03796/) (2021)
- [Multi-agent Soft Actor-Critic Based Hybrid Motion Planner for Mobile Robots](https://mlsanity.com/paper/2112.06594/) (2021)
- [Effective Multi-Agent Deep Reinforcement Learning Control with Relative Entropy Regularization](https://mlsanity.com/paper/2309.14727/) (2023)
- [SACPlanner: Real-World Collision Avoidance with a Soft Actor Critic Local Planner and Polar State Representations](https://mlsanity.com/paper/2303.11801/) (2023)
- [Risk-anticipatory autonomous driving strategies considering vehicles' weights, based on hierarchical deep reinforcement learning](https://mlsanity.com/paper/2401.08661/) (2023)

## Built on by (most cited first, 1 of 1)

- [Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion](https://mlsanity.com/paper/2606.18469/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has shown great potential in complex control tasks, particularly when combined with deep neural networks within the Actor-Critic (AC) framework. However, in practical applications, balancing exploration, learning stability, and sample efficiency remains a significant challenge. Traditional methods such as Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) address these issues by incorporating entropy or relative entropy regularization, but often face problems of instability and low sample efficiency. In this paper, we propose the Conservative Soft Actor-Critic (CSAC) algorithm, which seamlessly integrates entropy and relative entropy regularization within the AC framework. CSAC improves exploration through entropy regularization while avoiding overly aggressive policy updates with the use of relative entropy regularization. Evaluations on benchmark tasks and real-world robotic simulations demonstrate that CSAC offers significant improvements in stability and efficiency over existing methods. These findings suggest that CSAC provides strong robustness and application potential in control tasks under dynamic environments.