# Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/1908.03761 · HTML: https://mlsanity.com/paper/1908.03761/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2019.

## Built on by (most cited first, 30 of 33)

- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)](https://mlsanity.com/paper/2109.04024/) (2021)
- [Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Survey](https://mlsanity.com/paper/2504.21048/) (2025)
- [FoX: Formation-aware exploration in multi-agent reinforcement learning](https://mlsanity.com/paper/2308.11272/) (2023)
- [ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control](https://mlsanity.com/paper/2111.08067/) (2021)
- [Mean-Field Control based Approximation of Multi-Agent Reinforcement Learning in Presence of a Non-decomposable Shared Global State](https://mlsanity.com/paper/2301.06889/) (2023)
- [DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback](https://mlsanity.com/paper/2306.07553/) (2023)
- [Multi-Agent Reinforcement Learning Based on Representational Communication for Large-Scale Traffic Signal Control](https://mlsanity.com/paper/2310.02435/) (2023)
- [Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?](https://mlsanity.com/paper/2203.00035/) (2022)
- [Large-Scale Traffic Signal Control Using Constrained Network Partition and Adaptive Deep Reinforcement Learning](https://mlsanity.com/paper/2303.11899/) (2023)
- [Mean-Field Approximation of Cooperative Constrained Multi-Agent Reinforcement Learning (CMARL)](https://mlsanity.com/paper/2209.07437/) (2022)
- [Towards Multi-agent Reinforcement Learning based Traffic Signal Control through Spatio-temporal Hypergraphs](https://mlsanity.com/paper/2404.11014/) (2024)
- [Multi-intersection Traffic Optimisation: A Benchmark Dataset and a Strong Baseline](https://mlsanity.com/paper/2101.09640/) (2021)
- [On the Near-Optimality of Local Policies in Large Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03491/) (2022)
- [Large-Scale Traffic Signal Control by a Nash Deep Q-network Approach](https://mlsanity.com/paper/2301.00637/) (2023)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [Comparing Reinforcement Learning and Human Learning using the Game of Hidden Rules](https://mlsanity.com/paper/2306.17766/) (2023)
- [ROMA-iQSS: An Objective Alignment Approach via State-Based Value Learning and ROund-Robin Multi-Agent Scheduling](https://mlsanity.com/paper/2404.03984/) (2024)
- [Traffic Co-Simulation Framework Empowered by Infrastructure Camera Sensing and Reinforcement Learning](https://mlsanity.com/paper/2412.03925/) (2024)
- [Federated Hierarchical Reinforcement Learning for Adaptive Traffic Signal Control](https://mlsanity.com/paper/2504.05553/) (2025)
- [Human-Centric Traffic Signal Control for Equity: A Multi-Agent Action Branching Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2602.02959/) (2026)
- [Scalable Traffic Signal Controls using Fog-Cloud Based Multiagent Reinforcement Learning](https://mlsanity.com/paper/2110.05564/) (2021)
- [A Reinforcement Learning-based Adaptive Control Model for Future Street Planning, An Algorithm and A Case Study](https://mlsanity.com/paper/2112.05434/) (2021)
- [CGIBNet: Bandwidth-constrained Communication with Graph Information Bottleneck in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2112.10374/) (2021)
- [A Decentralized Communication Framework based on Dual-Level Recurrence for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.10612/) (2022)
- [Traffic Shaping and Hysteresis Mitigation Using Deep Reinforcement Learning in a Connected Driving Environment](https://mlsanity.com/paper/2302.03141/) (2023)
- [Heterogeneous Multi-Agent Reinforcement Learning via Mirror Descent Policy Optimization](https://mlsanity.com/paper/2308.06741/) (2023)
- [Toward Dependency Dynamics in Multi-Agent Reinforcement Learning for Traffic Signal Control](https://mlsanity.com/paper/2502.16608/) (2025)
- [Fully Decentralized Cooperative Multi-Agent Reinforcement Learning is A Context Modeling Problem](https://mlsanity.com/paper/2509.15519/) (2025)
- [Spatio-temporal dual-stage hypergraph MARL for human-centric multimodal corridor traffic signal control](https://mlsanity.com/paper/2602.17068/) (2026)

## Abstract (arXiv)

Finding the optimal signal timing strategy is a difficult task for the problem of large-scale traffic signal control (TSC). Multi-Agent Reinforcement Learning (MARL) is a promising method to solve this problem. However, there is still room for improvement in extending to large-scale problems and modeling the behaviors of other agents for each individual agent. In this paper, a new MARL, called Cooperative double Q-learning (Co-DQL), is proposed, which has several prominent features. It uses a highly scalable independent double Q-learning method based on double estimators and the UCB policy, which can eliminate the over-estimation problem existing in traditional independent Q-learning while ensuring exploration. It uses mean field approximation to model the interaction among agents, thereby making agents learn a better cooperative strategy. In order to improve the stability and robustness of the learning process, we introduce a new reward allocation mechanism and a local state sharing method. In addition, we analyze the convergence properties of the proposed algorithm. Co-DQL is applied on TSC and tested on a multi-traffic signal simulator. According to the results obtained on several traffic scenarios, Co- DQL outperforms several state-of-the-art decentralized MARL algorithms. It can effectively shorten the average waiting time of the vehicles in the whole road system.