# Enforcing Cooperative Safety for Reinforcement Learning-based Mixed-Autonomy Platoon Control

arXiv: https://arxiv.org/abs/2411.10031 · HTML: https://mlsanity.com/paper/2411.10031/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multi-agent reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 59 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [Multi-agent reinforcement learning](https://mlsanity.com/deeprl/multi-agent-reinforcement-learning/)

## Builds on (most cited first, 21 of 21)

- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Planning in Dynamic Environments using Conformal Prediction](https://mlsanity.com/paper/2210.10254/) (2022)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [Data-Enabled Predictive Control: In the Shallows of the DeePC](https://mlsanity.com/paper/1811.05890/) (2018)
- [Safe Reinforcement Learning Using Black-Box Reachability Analysis](https://mlsanity.com/paper/2204.07417/) (2022)
- [Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2410.24152/) (2024)
- [DeeP-LCC: Data-EnablEd Predictive Leading Cruise Control in Mixed Traffic Flow](https://mlsanity.com/paper/2203.10639/) (2022)
- [Enhancing System-Level Safety in Mixed-Autonomy Platoon via Safe Reinforcement Learning](https://mlsanity.com/paper/2401.11148/) (2024)
- [Interaction-Aware Trajectory Prediction for Safe Motion Planning in Autonomous Driving: A Transformer-Transfer Learning Approach](https://mlsanity.com/paper/2411.01475/) (2024)
- [Leading Cruise Control in Mixed Traffic Flow: System Modeling, Controllability, and String Stability](https://mlsanity.com/paper/2012.04313/) (2020)
- [A Parameter Privacy-Preserving Strategy for Mixed-Autonomy Platoon Control](https://mlsanity.com/paper/2401.15561/) (2024)
- [Agile Decision-Making and Safety-Critical Motion Planning for Emergency Autonomous Vehicles](https://mlsanity.com/paper/2409.08665/) (2024)
- [Energy-efficient Connected Cruise Control with Lean Penetration of Connected Vehicles](https://mlsanity.com/paper/2205.03473/) (2022)
- [Decentralized Robust Data-driven Predictive Control for Smoothing Mixed Traffic Flow](https://mlsanity.com/paper/2401.15826/) (2024)
- [Towards Safe and Robust Autonomous Vehicle Platooning: A Self-Organizing Cooperative Control Framework](https://mlsanity.com/paper/2408.09468/) (2024)
- [Cumulative Flow Diagram-Based Fixed-Time Signal Timing Optimization at Isolated Intersections Using Connected Vehicle Trajectory Data](https://mlsanity.com/paper/2210.10508/) (2022)

## Abstract (arXiv)

It is recognized that the control of mixed-autonomy platoons comprising connected and automated vehicles (CAVs) and human-driven vehicles (HDVs) can enhance traffic flow. Among existing methods, Multi-Agent Reinforcement Learning (MARL) appears to be a promising control strategy because it can manage complex scenarios in real time. However, current research on MARL-based mixed-autonomy platoon control suffers from several limitations. First, existing MARL approaches address safety by penalizing safety violations in the reward function, thus lacking theoretical safety guarantees due to the black-box nature of RL. Second, few studies have explored the cooperative safety of multi-CAV platoons, where CAVs can be coordinated to further enhance the system-level safety involving the safety of both CAVs and HDVs. Third, existing work tends to make an unrealistic assumption that the behavior of HDVs and CAVs is publicly known and rationale. To bridge the research gaps, we propose a safe MARL framework for mixed-autonomy platoons. Specifically, this framework (i) characterizes cooperative safety by designing a cooperative Control Barrier Function (CBF), enabling CAVs to collaboratively improve the safety of the entire platoon, (ii) provides a safety guarantee to the MARL-based controller by integrating the CBF-based safety constraints into MARL through a differentiable quadratic programming (QP) layer, and (iii) incorporates a conformal prediction module that enables each CAV to estimate the unknown behaviors of the surrounding vehicles with uncertainty qualification. Simulation results show that our proposed control strategy can effectively enhance the system-level safety through CAV cooperation of a mixed-autonomy platoon with a minimal impact on control performance.