mlsanityData to 7 Oct 2026

Constrained Reinforcement Learning with Smoothed Log Barrier Function

arXiv 2403.14508 ↗First posted 21 Mar 2024cs.LG cs.AI cs.SYcited by 12

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) has been widely applied to many control tasks and substantially improved the performances compared to conventional control methods in many domains where the reward function is well defined. However, for many real-world problems, it is often more convenient to formulate optimization problems in terms of rewards and constraints simultaneously. Optimizing such constrained problems via reward shaping can be difficult as it requires tedious manual tuning of reward functions with several interacting terms. Recent formulations which include constraints mostly require a pre-training phase, which often needs human expertise to collect data or assumes having a sub-optimal policy readily available. We propose a new constrained RL method called CSAC-LB (Constrained Soft Actor-Critic with Log Barrier Function), which achieves competitive performance without any pre-training by applying a linear smoothed log barrier function to an additional safety critic. It implements an adaptive penalty for policy learning and alleviates the numerical issues that are known to complicate the application of the log barrier function method. As a result, we show that with CSAC-LB, we achieve state-of-the-art performance on several constrained control tasks with different levels of difficulty and evaluate our methods in a locomotion task on a real quadruped robot platform.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

33 of 54 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2021Convex optimizationcited by 2,479
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2017Constrained Policy Optimizationcited by 928
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  8. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  9. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  10. 2019Control Barrier Functions: Theory and Applicationscited by 332
  11. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  12. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
21 more
  1. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  2. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  3. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  4. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  5. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  6. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  7. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  8. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  9. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  10. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  11. 2021Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robotscited by 67
  12. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  13. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  14. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  15. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  16. 2019Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensionscited by 30
  17. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 25
  18. 2020Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEscited by 22
  19. 2015Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regretcited by 15
  20. 2017Safe Policy Search with Gaussian Process Modelscited by 9
  21. 2018A Logarithmic Barrier Method For Proximal Policy Optimizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  2. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
  3. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9
  4. 2025Central Path Proximal Policy Optimizationcited by 3
  5. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  6. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  7. 2025Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulationscited by 2
  8. 2024Constrained Reinforcement Learning for Safe Heat Pump Controlcited by 1
  9. 2025HoP: Homeomorphic Polar Learning for Hard Constrained Optimizationcited by 1
  10. 2025Analysis of approximate linear programming solution to Markov decision problem with log barrier functioncited by 1
  11. 2024Revisiting Safe Exploration in Safe Reinforcement learningcited by 0
  12. 2026CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learningcited by 0