# Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

arXiv: https://arxiv.org/abs/2105.05701 · HTML: https://mlsanity.com/paper/2105.05701/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2021.

## Built on by (most cited first, 30 of 41)

- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic](https://mlsanity.com/paper/2111.06318/) (2021)
- [KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2407.14239/) (2024)
- [Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic](https://mlsanity.com/paper/2202.00881/) (2022)
- [Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2410.24152/) (2024)
- [Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey](https://mlsanity.com/paper/2504.21048/) (2025)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [Safe Multi-Agent Reinforcement Learning with Bilevel Optimization in Autonomous Driving](https://mlsanity.com/paper/2405.18209/) (2024)
- [Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Driving](https://mlsanity.com/paper/2412.18511/) (2024)
- [Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications](https://mlsanity.com/paper/2306.06808/) (2023)
- [Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control](https://mlsanity.com/paper/2408.17380/) (2024)
- [BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systems](https://mlsanity.com/paper/2501.01593/) (2025)
- [Prediction-aware and Reinforcement Learning based Altruistic Cooperative Driving](https://mlsanity.com/paper/2211.10585/) (2022)
- [Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.10351/) (2023)
- [A Cascading Cooperative Multi-agent Framework for On-ramp Merging Control Integrating Large Language Models](https://mlsanity.com/paper/2503.08199/) (2025)
- [Large Language Models and Foundation Models in Smart Agriculture: Basics, Opportunities, and Challenges](https://mlsanity.com/paper/2308.06668/) (2023)
- [Safe, Efficient, Comfort, and Energy-saving Automated Driving through Roundabout Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2306.11465/) (2023)
- [Trust-MARL: Trust-Based Multi-Agent Reinforcement Learning Framework for Cooperative On-Ramp Merging Control in Heterogeneous Traffic Flow](https://mlsanity.com/paper/2506.12600/) (2025)
- [Multi-Agent Reinforcement Learning for Joint Police Patrol and Dispatch](https://mlsanity.com/paper/2409.02246/) (2024)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Learning-based social coordination to improve safety and robustness of cooperative autonomous vehicles in mixed traffic](https://mlsanity.com/paper/2211.11963/) (2022)
- [Learning to Change: Choreographing Mixed Traffic Through Lateral Control and Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2403.14879/) (2024)
- [Reinforcement Learning for Freeway Lane-Change Regulation via Connected Vehicles](https://mlsanity.com/paper/2412.04341/) (2024)
- [Evaluating Scenario-based Decision-making for Interactive Autonomous Driving Using Rational Criteria: A Survey](https://mlsanity.com/paper/2501.01886/) (2025)
- [A Differentiated Reward Method for Reinforcement Learning based Multi-Vehicle Cooperative Decision-Making Algorithms](https://mlsanity.com/paper/2502.00352/) (2025)
- [Safe and Efficient CAV Lane Changing using Decentralised Safety Shields](https://mlsanity.com/paper/2505.01453/) (2025)
- [TGLD: A Trust-Aware Game-Theoretic Lane-Changing Decision Framework for Automated Vehicles in Heterogeneous Traffic](https://mlsanity.com/paper/2507.10075/) (2025)
- [An Evolving Scenario Generation Method based on Dual-modal Driver Model Trained by Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.02027/) (2025)
- [Markov Potential Game and Multi-Agent Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2603.19188/) (2026)
- [Recent Progress in Energy Management of Connected Hybrid Electric Vehicles Using Reinforcement Learning](https://mlsanity.com/paper/2308.14602/) (2023)

## Abstract (arXiv)

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.