Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint
Not on the map. Why?
Abstract (arXiv)
Soft actor-critic (SAC) in reinforcement learning is expected to be one of the next-generation robot control schemes. Its ability to maximize policy entropy would make a robotic controller robust to noise and perturbation, which is useful for real-world robot applications. However, the priority of maximizing the policy entropy is automatically tuned in the current implementation, the rule of which can be interpreted as one for equality constraint, binding the policy entropy into its specified lower bound. The current SAC is therefore no longer maximize the policy entropy, contrary to our expectation. To resolve this issue in SAC, this paper improves its implementation with a learnable state-dependent slack variable for appropriately handling the inequality constraint to maximize the policy entropy by reformulating it as the corresponding equality constraint. The introduced slack variable is optimized by a switching-type loss function that takes into account the dual objectives of satisfying the equality constraint and checking the lower bound. In Mujoco and Pybullet simulators, the modified SAC statistically achieved the higher robustness for adversarial attacks than before while regularizing the norm of action. A real-robot variable impedance task was demonstrated for showing the applicability of the modified SAC to real-world robot control. In particular, the modified SAC maintained adaptive behaviors for physical human-robot interaction, which had no experience at all during training. https://youtu.be/EH3xVtlVaJw
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 8 Mar 2023.
Builds on
21 of 40 references.
References · 21
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2021Convex optimizationcited by 2,479
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2019Root Mean Square Layer Normalizationcited by 1,425
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2018Learning Dexterous In-Hand Manipulationcited by 643
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
- 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
- 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
- 2019Soft Actor-Critic for Discrete Action Settingscited by 161
- 2021Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robotscited by 95
9 more
- 2019Simulation-based reinforcement learning for real-world autonomous drivingcited by 51
- 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 37
- 2016Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangiancited by 29
- 2022L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learningcited by 22
- 2021Squareplus: A Softplus-Like Algebraic Rectifiercited by 18
- 2020Meta-SAC: Auto-tune the Entropy Temperature of Soft Actor-Critic via Metagradientcited by 14
- 2021Target Entropy Annealing for Discrete Soft Actor-Criticcited by 10
- 2022Consolidated Adaptive T-soft Update for Deep Reinforcement Learningcited by 5
- 2021Mirror-Descent Inverse Kinematics for Box-constrained Joint Spacecited by 1
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.