# Reinforcement learning for traffic signal control in hybrid action space

arXiv: https://arxiv.org/abs/2211.12956 · HTML: https://mlsanity.com/paper/2211.12956/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 76 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control](https://mlsanity.com/paper/1903.04527/) (2019)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [CoLight: Learning Network-level Cooperation for Traffic Signal Control](https://mlsanity.com/paper/1905.05717/) (2019)
- [Learning Phase Competition for Traffic Signal Control](https://mlsanity.com/paper/1905.04722/) (2019)
- [Reinforcement Learning with Parameterized Actions](https://mlsanity.com/paper/1509.01644/) (2015)
- [AttendLight: Universal Attention-Based Reinforcement Learning Model for Traffic Signal Control](https://mlsanity.com/paper/2010.05772/) (2020)
- [Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space](https://mlsanity.com/paper/1810.06394/) (2018)
- [Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space](https://mlsanity.com/paper/1903.01344/) (2019)
- [Deep Reinforcement Learning for Traffic Light Control in Vehicular Networks](https://mlsanity.com/paper/1803.11115/) (2018)
- [HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation](https://mlsanity.com/paper/2109.05490/) (2021)
- [Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning](https://mlsanity.com/paper/1704.08883/) (2017)
- [Deep Deterministic Policy Gradient for Urban Traffic Light Control](https://mlsanity.com/paper/1703.09035/) (2017)
- [Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces](https://mlsanity.com/paper/1905.04388/) (2019)
- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [CVLight: Decentralized Learning for Adaptive Traffic Signal Control with Connected Vehicles](https://mlsanity.com/paper/2104.10340/) (2021)
- [Using Reinforcement Learning with Partial Vehicle Detection for Intelligent Traffic Signal Control](https://mlsanity.com/paper/1807.01628/) (2018)
- [Policy Regularization via Noisy Advantage Values for Cooperative Multi-agent Actor-Critic methods](https://mlsanity.com/paper/2106.14334/) (2021)

## Built on by (most cited first, 5 of 5)

- [DRL-Based Optimization for AoI and Energy Consumption in C-V2X Enabled IoV](https://mlsanity.com/paper/2411.13104/) (2024)
- [Human-Centric Traffic Signal Control for Equity: A Multi-Agent Action Branching Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2602.02959/) (2026)
- [A Parallel Hybrid Action Space Reinforcement Learning Model for Real-world Adaptive Traffic Signal Control](https://mlsanity.com/paper/2503.14250/) (2025)
- [Spatio-temporal dual-stage hypergraph MARL for human-centric multimodal corridor traffic signal control](https://mlsanity.com/paper/2602.17068/) (2026)
- [Temporal Action Representation Learning for Tactical Resource Control and Subsequent Maneuver Generation](https://mlsanity.com/paper/2602.18716/) (2026)

## Abstract (arXiv)

The prevailing reinforcement-learning-based traffic signal control methods are typically staging-optimizable or duration-optimizable, depending on the action spaces. In this paper, we propose a novel control architecture, TBO, which is based on hybrid proximal policy optimization. To the best of our knowledge, TBO is the first RL-based algorithm to implement synchronous optimization of the staging and duration. Compared to discrete and continuous action spaces, hybrid action space is a merged search space, in which TBO better implements the trade-off between frequent switching and unsaturated release. Experiments are given to demonstrate that TBO reduces the queue length and delay by 13.78% and 14.08% on average, respectively, compared to the existing baselines. Furthermore, we calculate the Gini coefficients of the right-of-way to indicate TBO does not harm fairness while improving efficiency.