# Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic

arXiv: https://arxiv.org/abs/2111.06318 · HTML: https://mlsanity.com/paper/2111.06318/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Cooperative and V2X driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Placed in

- End-to-end autonomous driving / [Cooperative and V2X driving](https://mlsanity.com/driving/cooperative-and-v2x-driving/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control](https://mlsanity.com/paper/1903.04527/) (2019)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1911.04175/) (2019)
- [Multi-agent Reinforcement Learning for Networked System Control](https://mlsanity.com/paper/2004.01339/) (2020)
- [Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic](https://mlsanity.com/paper/2105.05701/) (2021)
- [Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.10056/) (2018)
- [Autonomous Driving using Safe Reinforcement Learning by Incorporating a Regret-based Human Lane-Changing Decision Model](https://mlsanity.com/paper/1910.04803/) (2019)
- [A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network](https://mlsanity.com/paper/2010.05437/) (2020)
- [Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm](https://mlsanity.com/paper/1906.02275/) (2019)
- [Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion](https://mlsanity.com/paper/2010.05436/) (2020)
- [A Cooperative Control Framework for CAV Lane Change in a Mixed Traffic Environment](https://mlsanity.com/paper/2010.05439/) (2020)
- [Parameter Sharing Reinforcement Learning Architecture for Multi Agent Driving Behaviors](https://mlsanity.com/paper/1811.07214/) (2018)
- [Efficient Motion Planning for Automated Lane Change based on Imitation Learning and Mixed-Integer Optimization](https://mlsanity.com/paper/1904.08784/) (2019)
- [Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning](https://mlsanity.com/paper/2106.07892/) (2021)

## Built on by (most cited first, 20 of 20)

- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priors](https://mlsanity.com/paper/2408.03200/) (2024)
- [AgentGit: A Version Control Framework for Reliable and Scalable LLM-Powered Multi-Agent Systems](https://mlsanity.com/paper/2511.00628/) (2025)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [A Constrained Multi-Agent Reinforcement Learning Approach to Autonomous Traffic Signal Control](https://mlsanity.com/paper/2503.23626/) (2025)
- [GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent Systems](https://mlsanity.com/paper/2602.15776/) (2026)
- [Hiding in Plain Sight: Differential Privacy Noise Exploitation for Evasion-resilient Localized Poisoning Attacks in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2307.00268/) (2023)
- [Learning to Model Diverse Driving Behaviors in Highly Interactive Autonomous Driving Scenarios with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2402.13481/) (2024)
- [Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Survey](https://mlsanity.com/paper/2508.20315/) (2025)
- [BRNES: Enabling Security and Privacy-aware Experience Sharing in Multiagent Robotic and Autonomous Systems](https://mlsanity.com/paper/2308.01274/) (2023)
- [Mix Q-learning for Lane Changing: A Collaborative Decision-Making Method in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2406.09755/) (2024)
- [Safe and Efficient CAV Lane Changing using Decentralised Safety Shields](https://mlsanity.com/paper/2505.01453/) (2025)
- [Pedestrian-Aware LLM-Driven Behavioral Planning for Autonomous Vehicles](https://mlsanity.com/paper/2605.16858/) (2026)
- [Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning](https://mlsanity.com/paper/2211.16069/) (2022)
- [Recent Progress in Energy Management of Connected Hybrid Electric Vehicles Using Reinforcement Learning](https://mlsanity.com/paper/2308.14602/) (2023)
- [Counterfactual experience augmented off-policy reinforcement learning](https://mlsanity.com/paper/2503.13842/) (2025)
- [Multi-Agent Coordination in Autonomous Vehicle Routing: A Simulation-Based Study of Communication, Memory, and Routing Loops](https://mlsanity.com/paper/2511.17656/) (2025)
- [A Collaborative Safety Shield for Safe and Efficient CAV Lane Changes in Congested On-Ramp Merging](https://mlsanity.com/paper/2602.10007/) (2026)
- [Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.04492/) (2026)
- [Reinforcement Learning-Based Control of CAV Platoon Joining Maneuvers in Mixed Traffic](https://mlsanity.com/paper/2608.26860/) (2026)

## Abstract (arXiv)

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL), a powerful data-driven control method, has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic network (MA2C) is developed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is proposed to incorporate fuel efficiency, driving comfort, and safety of autonomous driving. Comprehensive experimental results, conducted under three different traffic densities and various levels of human driver aggressiveness, show that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety and driver comfort.