# FISAR: Forward Invariant Safe Reinforcement Learning with a Deep Neural Network-Based Optimize

arXiv: https://arxiv.org/abs/2006.11419 · HTML: https://mlsanity.com/paper/2006.11419/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 5 of 5)

- [SABLAS: Learning Safe Control for Black-box Dynamical Systems](https://mlsanity.com/paper/2201.01918/) (2022)
- [Constrained Meta-Reinforcement Learning for Adaptable Safety Guarantee with Differentiable Convex Programming](https://mlsanity.com/paper/2312.10230/) (2023)
- [Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programming](https://mlsanity.com/paper/2310.02459/) (2023)
- [FORESEE: Prediction with Expansion-Compression Unscented Transform for Online Policy Optimization](https://mlsanity.com/paper/2209.12644/) (2022)
- [Towards an Adaptable and Generalizable Optimization Engine in Decision and Control: A Meta Reinforcement Learning Approach](https://mlsanity.com/paper/2401.02508/) (2024)

## Abstract (arXiv)

This paper investigates reinforcement learning with constraints, which are indispensable in safety-critical environments. To drive the constraint violation monotonically decrease, we take the constraints as Lyapunov functions and impose new linear constraints on the policy parameters' updating dynamics. As a result, the original safety set can be forward-invariant. However, because the new guaranteed-feasible constraints are imposed on the updating dynamics instead of the original policy parameters, classic optimization algorithms are no longer applicable. To address this, we propose to learn a generic deep neural network (DNN)-based optimizer to optimize the objective while satisfying the linear constraints. The constraint-satisfaction is achieved via projection onto a polytope formulated by multiple linear inequality constraints, which can be solved analytically with our newly designed metric. To the best of our knowledge, this is the textitfirst DNN-based optimizer for constrained optimization with the forward invariance guarantee. We show that our optimizer trains a policy to decrease the constraint violation and maximize the cumulative reward monotonically. Results on numerical constrained optimization and obstacle-avoidance navigation validate the theoretical findings.