# Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization

arXiv: https://arxiv.org/abs/1909.10651 · HTML: https://mlsanity.com/paper/1909.10651/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 12 of 12)

- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.06042/) (2021)
- [Model-based graph reinforcement learning for inductive traffic signal control](https://mlsanity.com/paper/2208.00659/) (2022)
- [Deep Reinforcement Learning from Hierarchical Preference Design](https://mlsanity.com/paper/2309.02632/) (2023)
- [Attacking Deep Reinforcement Learning-Based Traffic Signal Control Systems with Colluding Vehicles](https://mlsanity.com/paper/2111.02845/) (2021)
- [A Critical Review of Traffic Signal Control and A Novel Unified View of Reinforcement Learning and Model Predictive Control Approaches for Adaptive Traffic Signal Control](https://mlsanity.com/paper/2211.14426/) (2022)
- [Real-Time Network-Level Traffic Signal Control: An Explicit Multiagent Coordination Method](https://mlsanity.com/paper/2306.08843/) (2023)
- [Never Explore Repeatedly in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.09909/) (2023)
- [Mix Q-learning for Lane Changing: A Collaborative Decision-Making Method in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2406.09755/) (2024)
- [TransferLight: Zero-Shot Traffic Signal Control on any Road-Network](https://mlsanity.com/paper/2412.09719/) (2024)
- [Smart Traffic Signals: Comparing MARL and Fixed-Time Strategies](https://mlsanity.com/paper/2505.14544/) (2025)
- [Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward](https://mlsanity.com/paper/2605.24992/) (2026)

## Abstract (arXiv)

Traffic congestion in metropolitan areas is a world-wide problem that can be ameliorated by traffic lights that respond dynamically to real-time conditions. Recent studies applying deep reinforcement learning (RL) to optimize single traffic lights have shown significant improvement over conventional control. However, optimization of global traffic condition over a large road network fundamentally is a cooperative multi-agent control problem, for which single-agent RL is not suitable due to environment non-stationarity and infeasibility of optimizing over an exponential joint-action space. Motivated by these challenges, we propose QCOMBO, a simple yet effective multi-agent reinforcement learning (MARL) algorithm that combines the advantages of independent and centralized learning. We ensure scalability by selecting actions from individually optimized utility functions, which are shaped to maximize global performance via a novel consistency regularization loss between individual utility and a global action-value function. Experiments on diverse road topologies and traffic flow conditions in the SUMO traffic simulator show competitive performance of QCOMBO versus recent state-of-the-art MARL algorithms. We further show that policies trained on small sub-networks can effectively generalize to larger networks under different traffic flow conditions, providing empirical evidence for the suitability of MARL for intelligent traffic control.