# Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety

arXiv: https://arxiv.org/abs/2105.10682 · HTML: https://mlsanity.com/paper/2105.10682/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 22 May 2021.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [Deep Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1811.11329/) (2018)
- [Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function](https://mlsanity.com/paper/2103.01556/) (2021)
- [Open-Sourced Reinforcement Learning Environments for Surgical Robotics](https://mlsanity.com/paper/1903.02090/) (2019)
- [A Review of Reinforcement Learning for Autonomous Building Energy Management](https://mlsanity.com/paper/1903.05196/) (2019)
- [Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning](https://mlsanity.com/paper/2102.08539/) (2021)
- [Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints](https://mlsanity.com/paper/1911.11397/) (2019)
- [Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic](https://mlsanity.com/paper/2002.05502/) (2020)
- [Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and model](https://mlsanity.com/paper/2102.11513/) (2021)

## Built on by (most cited first, 30 of 30)

- [Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model](https://mlsanity.com/paper/2401.10700/) (2024)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [Iterative Reachability Estimation for Safe Reinforcement Learning](https://mlsanity.com/paper/2309.13528/) (2023)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks](https://mlsanity.com/paper/2212.05727/) (2022)
- [Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate](https://mlsanity.com/paper/2210.07553/) (2022)
- [Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2407.09645/) (2024)
- [Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach](https://mlsanity.com/paper/2304.04066/) (2023)
- [GUARD: A Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2305.13681/) (2023)
- [State-wise Constrained Policy Optimization](https://mlsanity.com/paper/2306.12594/) (2023)
- [SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving](https://mlsanity.com/paper/2206.08528/) (2022)
- [Verified Probabilistic Policies for Deep Reinforcement Learning](https://mlsanity.com/paper/2201.03698/) (2022)
- [Reduced Policy Optimization for Continuous Control with Hard Constraints](https://mlsanity.com/paper/2310.09574/) (2023)
- [Action Mapping for Reinforcement Learning in Continuous Environments with Constraints](https://mlsanity.com/paper/2412.04327/) (2024)
- [Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.12953/) (2021)
- [Conservative Distributional Reinforcement Learning with Safety Constraints](https://mlsanity.com/paper/2201.07286/) (2022)
- [Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2403.18209/) (2024)
- [Stable and Safe Human-aligned Reinforcement Learning through Neural Ordinary Differential Equations](https://mlsanity.com/paper/2401.13148/) (2024)
- [FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning](https://mlsanity.com/paper/2412.08880/) (2024)
- [A Multiplicative Value Function for Safe and Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.04118/) (2023)
- [Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints](https://mlsanity.com/paper/2411.19193/) (2024)
- [Neuro-symbolic Action Masking for Deep Reinforcement Learning](https://mlsanity.com/paper/2602.10598/) (2026)
- [Augmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learning](https://mlsanity.com/paper/2605.00667/) (2026)
- [Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction](https://mlsanity.com/paper/2410.01212/) (2024)
- [Barrier Function Overrides For Non-Convex Fixed Wing Flight Control and Self-Driving Cars](https://mlsanity.com/paper/2505.05548/) (2025)
- [Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving](https://mlsanity.com/paper/2512.03774/) (2025)
- [Conformal Reachability for Safe Control in Unknown Environments](https://mlsanity.com/paper/2602.03799/) (2026)
- [A Safety-Constrained Reinforcement Learning Framework for Reliable Wireless Autonomy](https://mlsanity.com/paper/2602.13207/) (2026)
- [A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving](https://mlsanity.com/paper/2609.09650/) (2026)
- [Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learning](https://mlsanity.com/paper/2609.33337/) (2026)

## Abstract (arXiv)

The safety constraints commonly used by existing safe reinforcement learning (RL) methods are defined only on expectation of initial states, but allow each certain state to be unsafe, which is unsatisfying for real-world safety-critical tasks. In this paper, we introduce the feasible actor-critic (FAC) algorithm, which is the first model-free constrained RL method that considers statewise safety, e.g, safety for each initial state. We claim that some states are inherently unsafe no matter what policy we choose, while for other states there exist policies ensuring safety, where we say such states and policies are feasible. By constructing a statewise Lagrange function available on RL sampling and adopting an additional neural network to approximate the statewise Lagrange multiplier, we manage to obtain the optimal feasible policy which ensures safety for each feasible state and the safest possible policy for infeasible states. Furthermore, the trained multiplier net can indicate whether a given state is feasible or not through the statewise complementary slackness condition. We provide theoretical guarantees that FAC outperforms previous expectation-based constrained RL methods in terms of both constraint satisfaction and reward optimization. Experimental results on both robot locomotive tasks and safe exploration tasks verify the safety enhancement and feasibility interpretation of the proposed method.