# Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization

arXiv: https://arxiv.org/abs/2003.01303 · HTML: https://mlsanity.com/paper/2003.01303/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 30 of 35)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [CAT: Closed-loop Adversarial Training for Safe End-to-End Driving](https://mlsanity.com/paper/2310.12432/) (2023)
- [Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach](https://mlsanity.com/paper/2109.06332/) (2021)
- [Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path](https://mlsanity.com/paper/2206.02678/) (2022)
- [Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm](https://mlsanity.com/paper/2206.05850/) (2022)
- [Learning Adversarial MDPs with Stochastic Hard Constraints](https://mlsanity.com/paper/2403.03672/) (2024)
- [Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback](https://mlsanity.com/paper/2307.02842/) (2023)
- [Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarios](https://mlsanity.com/paper/2210.02300/) (2022)
- [Optimal Strong Regret and Violation in Constrained MDPs via Policy Optimization](https://mlsanity.com/paper/2410.02275/) (2024)
- [RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes](https://mlsanity.com/paper/2405.04714/) (2024)
- [A Best-of-Both-Worlds Algorithm for Constrained MDPs with Long-Term Constraints](https://mlsanity.com/paper/2304.14326/) (2023)
- [Multi-task Safe Reinforcement Learning for Navigating Intersections in Dense Traffic](https://mlsanity.com/paper/2202.09644/) (2022)
- [SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving](https://mlsanity.com/paper/2206.08528/) (2022)
- [Provably Efficient CVaR RL in Low-rank MDPs](https://mlsanity.com/paper/2311.11965/) (2023)
- [Provably Correct Training of Neural Network Controllers Using Reachability Analysis](https://mlsanity.com/paper/2102.10806/) (2021)
- [Learning Constrained Markov Decision Processes With Non-stationary Rewards and Constraints](https://mlsanity.com/paper/2405.14372/) (2024)
- [Confident Natural Policy Gradient for Local Planning in qπ-realizable Constrained MDPs](https://mlsanity.com/paper/2406.18529/) (2024)
- [Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization](https://mlsanity.com/paper/2108.08448/) (2021)
- [Regret Analysis of Unichain Average Reward Constrained MDPs with General Parameterization](https://mlsanity.com/paper/2602.08000/) (2026)
- [Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning](https://mlsanity.com/paper/2104.08876/) (2021)
- [Safe Reinforcement Learning with Contrastive Risk Prediction](https://mlsanity.com/paper/2209.09648/) (2022)
- [Improve Generalization of Driving Policy at Signalized Intersections with Adversarial Learning](https://mlsanity.com/paper/2204.04403/) (2022)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [BAP-SRL: Bayesian Adaptive Priority Safe Reinforcement Learning for Vehicle Motion Planning at Mixed Traffic Intersections](https://mlsanity.com/paper/2601.21679/) (2026)
- [A Safety-Critical Decision Making and Control Framework Combining Machine Learning and Rule-based Algorithms](https://mlsanity.com/paper/2201.12819/) (2022)
- [A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot](https://mlsanity.com/paper/2409.14867/) (2024)
- [Pre-Trained Vision Models as Perception Backbones for Safety Filters in Autonomous Driving](https://mlsanity.com/paper/2410.22585/) (2024)
- [Assuring the Safety of Reinforcement Learning Components: AMLAS-RL](https://mlsanity.com/paper/2507.08848/) (2025)
- [Beyond Slater's Condition in Online CMDPs with Stochastic and Adversarial Constraints](https://mlsanity.com/paper/2509.20114/) (2025)
- [How to ensure a safe control strategy? Towards a SRL for urban transit autonomous operation](https://mlsanity.com/paper/2311.14457/) (2023)

## Abstract (arXiv)

Reinforcement learning (RL) is attracting increasing interests in autonomous driving due to its potential to solve complex classification and control problems. However, existing RL algorithms are rarely applied to real vehicles for two predominant problems: behaviours are unexplainable, and they cannot guarantee safety under new scenarios. This paper presents a safe RL algorithm, called Parallel Constrained Policy Optimization (PCPO), for two autonomous driving tasks. PCPO extends today's common actor-critic architecture to a three-component learning framework, in which three neural networks are used to approximate the policy function, value function and a newly added risk function, respectively. Meanwhile, a trust region constraint is added to allow large update steps without breaking the monotonic improvement condition. To ensure the feasibility of safety constrained problems, synchronized parallel learners are employed to explore different state spaces, which accelerates learning and policy-update. The simulations of two scenarios for autonomous vehicles confirm we can ensure safety while achieving fast learning.