# Large-Scale Traffic Signal Control by a Nash Deep Q-network Approach

arXiv: https://arxiv.org/abs/2301.00637 · HTML: https://mlsanity.com/paper/2301.00637/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 23 references that are in the map.
- First posted on arXiv 2 Jan 2023.

## Builds on (most cited first, 5 of 5)

- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control](https://mlsanity.com/paper/1903.04527/) (2019)
- [Deep Reinforcement Learning for Traffic Light Control in Vehicular Networks](https://mlsanity.com/paper/1803.11115/) (2018)
- [Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1908.03761/) (2019)
- [Deep Deterministic Policy Gradient for Urban Traffic Light Control](https://mlsanity.com/paper/1703.09035/) (2017)

## Built on by (most cited first, 4 of 4)

- [A Constrained Multi-Agent Reinforcement Learning Approach to Autonomous Traffic Signal Control](https://mlsanity.com/paper/2503.23626/) (2025)
- [Communication Strategy on Macro-and-Micro Traffic State in Cooperative Deep Reinforcement Learning for Regional Traffic Signal Control](https://mlsanity.com/paper/2502.13248/) (2025)
- [Toward Dependency Dynamics in Multi-Agent Reinforcement Learning for Traffic Signal Control](https://mlsanity.com/paper/2502.16608/) (2025)
- [A Hierarchical Deep Reinforcement Learning Framework for Traffic Signal Control with Predictable Cycle Planning](https://mlsanity.com/paper/2509.03118/) (2025)

## Abstract (arXiv)

Reinforcement Learning (RL) is currently one of the most commonly used techniques for traffic signal control (TSC), which can adaptively adjusted traffic signal phase and duration according to real-time traffic data. However, a fully centralized RL approach is beset with difficulties in a multi-network scenario because of exponential growth in state-action space with increasing intersections. Multi-agent reinforcement learning (MARL) can overcome the high-dimension problem by employing the global control of each local RL agent, but it also brings new challenges, such as the failure of convergence caused by the non-stationary Markov Decision Process (MDP). In this paper, we introduce an off-policy nash deep Q-Network (OPNDQN) algorithm, which mitigates the weakness of both fully centralized and MARL approaches. The OPNDQN algorithm solves the problem that traditional algorithms cannot be used in large state-action space traffic models by utilizing a fictitious game approach at each iteration to find the nash equilibrium among neighboring intersections, from which no intersection has incentive to unilaterally deviate. One of main advantages of OPNDQN is to mitigate the non-stationarity of multi-agent Markov process because it considers the mutual influence among neighboring intersections by sharing their actions. On the other hand, for training a large traffic network, the convergence rate of OPNDQN is higher than that of existing MARL approaches because it does not incorporate all state information of each agent. We conduct an extensive experiments by using Simulation of Urban MObility simulator (SUMO), and show the dominant superiority of OPNDQN over several existing MARL approaches in terms of average queue length, episode training reward and average waiting time.