# Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles

arXiv: https://arxiv.org/abs/2112.11217 · HTML: https://mlsanity.com/paper/2112.11217/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 79 references that are in the map.
- First posted on arXiv 18 Dec 2021.

## Builds on (most cited first, 30 of 35)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Provably Efficient Safe Exploration via Primal-Dual Policy Optimization](https://mlsanity.com/paper/2003.00534/) (2020)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping](https://mlsanity.com/paper/2011.02669/) (2020)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems](https://mlsanity.com/paper/1808.00924/) (2018)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Safe Policies for Reinforcement Learning via Primal-Dual Methods](https://mlsanity.com/paper/1911.09101/) (2019)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [Convergent Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/1910.12156/) (2019)
- [Safety-Critical Model Predictive Control with Discrete-Time Control Barrier Function](https://mlsanity.com/paper/2007.11718/) (2020)
- [Safe Reinforcement Learning Using Advantage-Based Intervention](https://mlsanity.com/paper/2106.09110/) (2021)
- [Safe Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2106.06239/) (2021)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)
- [Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function](https://mlsanity.com/paper/2103.01556/) (2021)
- [SAMBA: Safe Model-Based & Active Reinforcement Learning](https://mlsanity.com/paper/2006.09436/) (2020)
- [Model Predictive Contouring Control for Collision Avoidance in Unstructured Dynamic Environments](https://mlsanity.com/paper/2010.10190/) (2020)
- [Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning](https://mlsanity.com/paper/1612.02516/) (2016)
- [Safe Reinforcement Learning Using Robust Action Governor](https://mlsanity.com/paper/2102.10643/) (2021)

## Built on by (most cited first, 4 of 4)

- [Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC](https://mlsanity.com/paper/2412.19669/) (2024)
- [Learning Predictive Control with Deep Koopman Operators for Autonomous Vehicle Motion Planning](https://mlsanity.com/paper/2606.08136/) (2026)
- [A Nested Graph Reinforcement Learning-based Decision-making Strategy for Eco-platooning](https://mlsanity.com/paper/2408.07578/) (2024)
- [Learning-based control of a single-DOF Aero system](https://mlsanity.com/paper/2607.00640/) (2026)

## Abstract (arXiv)

Recently, safe reinforcement learning (RL) with the actor-critic structure for continuous control tasks has received increasing attention. It is still challenging to learn a near-optimal control policy with safety and convergence guarantees. Also, few works have addressed the safe RL algorithm design under time-varying safety constraints. This paper proposes a safe RL algorithm for optimal control of nonlinear systems with time-varying state and control constraints. In the proposed approach, we construct a novel barrier force-based control policy structure to guarantee control safety. A multi-step policy evaluation mechanism is proposed to predict the policy's safety risk under time-varying safety constraints and guide the policy to update safely. Theoretical results on stability and robustness are proven. Also, the convergence of the actor-critic implementation is analyzed. The performance of the proposed algorithm outperforms several state-of-the-art RL algorithms in the simulated Safety Gym environment. Furthermore, the approach is applied to the integrated path following and collision avoidance problem for two real-world intelligent vehicles. A differential-drive vehicle and an Ackermann-drive one are used to verify offline deployment and online learning performance, respectively. Our approach shows an impressive sim-to-real transfer capability and a satisfactory online control performance in the experiment.