# Multi-Constraint Safe Reinforcement Learning via Closed-form Solution for Log-Sum-Exp Approximation of Control Barrier Functions

arXiv: https://arxiv.org/abs/2505.00671 · HTML: https://mlsanity.com/paper/2505.00671/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 1 May 2025.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 21 of 21)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization](https://mlsanity.com/paper/1809.05504/) (2018)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [MIPaaL: Mixed Integer Program as a Layer](https://mlsanity.com/paper/1907.05912/) (2019)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [Safe Control Synthesis via Input Constrained Control Barrier Functions](https://mlsanity.com/paper/2104.01704/) (2021)
- [Control Barrier Functions and Input-to-State Safety with Application to Automated Vehicles](https://mlsanity.com/paper/2206.03568/) (2022)
- [Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanes](https://mlsanity.com/paper/2402.05279/) (2024)
- [Gradient Shaping for Multi-Constraint Safe Reinforcement Learning](https://mlsanity.com/paper/2312.15127/) (2023)
- [Learning Differentiable Safety-Critical Control using Control Barrier Functions for Generalization to Novel Environments](https://mlsanity.com/paper/2201.01347/) (2022)
- [Composing Control Barrier Functions for Complex Safety Specifications](https://mlsanity.com/paper/2309.06647/) (2023)
- [Episodic Learning for Safe Bipedal Locomotion with Control Barrier Functions and Projection-to-State Safety](https://mlsanity.com/paper/2105.01697/) (2021)
- [Compositions of Multiple Control Barrier Functions Under Input Constraints](https://mlsanity.com/paper/2210.01354/) (2022)
- [Multiple Control Barrier Functions: An Application to Reactive Obstacle Avoidance for a Multi-steering Tractor-trailer System](https://mlsanity.com/paper/2209.05156/) (2022)
- [Do No Harm: A Counterfactual Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/2405.11669/) (2024)

## Built on by (most cited first, 2 of 2)

- [DiffCVaR: Reinforcement Learning for Risk Adaptation via Differentiable CVaR Barrier Functions](https://mlsanity.com/paper/2605.21257/) (2026)
- [BarrierSteer: LLM Safety via Learning Barrier Steering](https://mlsanity.com/paper/2602.20102/) (2026)

## Abstract (arXiv)

The safety of training task policies and their subsequent application using reinforcement learning (RL) methods has become a focal point in the field of safe RL. A central challenge in this area remains the establishment of theoretical guarantees for safety during both the learning and deployment processes. Given the successful implementation of Control Barrier Function (CBF)-based safety strategies in a range of control-affine robotic systems, CBF-based safe RL demonstrates significant promise for practical applications in real-world scenarios. However, integrating these two approaches presents several challenges. First, embedding safety optimization within the RL training pipeline requires that the optimization outputs be differentiable with respect to the input parameters, a condition commonly referred to as differentiable optimization, which is non-trivial to solve. Second, the differentiable optimization framework confronts significant efficiency issues, especially when dealing with multi-constraint problems. To address these challenges, this paper presents a CBF-based safe RL architecture that effectively mitigates the issues outlined above. The proposed approach constructs a continuous AND logic approximation for the multiple constraints using a single composite CBF. By leveraging this approximation, a close-form solution of the quadratic programming is derived for the policy network in RL, thereby circumventing the need for differentiable optimization within the end-to-end safe RL pipeline. This strategy significantly reduces computational complexity because of the closed-form solution while maintaining safety guarantees. Simulation results demonstrate that, in comparison to existing approaches relying on differentiable optimization, the proposed method significantly reduces training computational costs while ensuring provable safety throughout the training process.