# Constrained Reinforcement Learning with Smoothed Log Barrier Function

arXiv: https://arxiv.org/abs/2403.14508 · HTML: https://mlsanity.com/paper/2403.14508/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2310.12567/) (2023)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research](https://mlsanity.com/paper/2305.09304/) (2023)
- [Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots](https://mlsanity.com/paper/2111.01674/) (2021)
- [Constrained Policy Optimization via Bayesian World Models](https://mlsanity.com/paper/2201.09802/) (2022)
- [SafeDreamer: Safe Reinforcement Learning with World Models](https://mlsanity.com/paper/2307.07176/) (2023)
- [Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk](https://mlsanity.com/paper/2206.04436/) (2022)
- [Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations](https://mlsanity.com/paper/2108.01846/) (2021)
- [Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensions](https://mlsanity.com/paper/1904.04205/) (2019)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs](https://mlsanity.com/paper/2009.01196/) (2020)

## Built on by (most cited first, 15 of 15)

- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [Embedding Safety into RL: A New Take on Trust Region Methods](https://mlsanity.com/paper/2411.02957/) (2024)
- [A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions](https://mlsanity.com/paper/2508.09128/) (2025)
- [Central Path Proximal Policy Optimization](https://mlsanity.com/paper/2506.00700/) (2025)
- [Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations](https://mlsanity.com/paper/2506.16016/) (2025)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory](https://mlsanity.com/paper/2405.19024/) (2024)
- [Constrained Reinforcement Learning for Safe Heat Pump Control](https://mlsanity.com/paper/2409.19716/) (2024)
- [HoP: Homeomorphic Polar Learning for Hard Constrained Optimization](https://mlsanity.com/paper/2502.00304/) (2025)
- [Analysis of approximate linear programming solution to Markov decision problem with log barrier function](https://mlsanity.com/paper/2509.19800/) (2025)
- [Revisiting Safe Exploration in Safe Reinforcement learning](https://mlsanity.com/paper/2409.01245/) (2024)
- [GeCCo -- a Generalist Contact-Conditioned Policy for Loco-Manipulation Skills on Legged Robots](https://mlsanity.com/paper/2509.17582/) (2025)
- [CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning](https://mlsanity.com/paper/2604.23576/) (2026)
- [Model-Based Reinforcement Learning with Double Oracle Efficiency in Policy Optimization and Offline Estimation](https://mlsanity.com/paper/2605.00393/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has been widely applied to many control tasks and substantially improved the performances compared to conventional control methods in many domains where the reward function is well defined. However, for many real-world problems, it is often more convenient to formulate optimization problems in terms of rewards and constraints simultaneously. Optimizing such constrained problems via reward shaping can be difficult as it requires tedious manual tuning of reward functions with several interacting terms. Recent formulations which include constraints mostly require a pre-training phase, which often needs human expertise to collect data or assumes having a sub-optimal policy readily available. We propose a new constrained RL method called CSAC-LB (Constrained Soft Actor-Critic with Log Barrier Function), which achieves competitive performance without any pre-training by applying a linear smoothed log barrier function to an additional safety critic. It implements an adaptive penalty for policy learning and alleviates the numerical issues that are known to complicate the application of the log barrier function method. As a result, we show that with CSAC-LB, we achieve state-of-the-art performance on several constrained control tasks with different levels of difficulty and evaluate our methods in a locomotion task on a real quadruped robot platform.