# Unified Automatic Control of Vehicular Systems with Reinforcement Learning

arXiv: https://arxiv.org/abs/2208.00268 · HTML: https://mlsanity.com/paper/2208.00268/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 71 references that are in the map.
- First posted on arXiv 30 Jul 2022.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [CoLight: Learning Network-level Cooperation for Traffic Signal Control](https://mlsanity.com/paper/1905.05717/) (2019)
- [Flow: A Modular Learning Framework for Mixed Autonomy Traffic](https://mlsanity.com/paper/1710.05465/) (2017)
- [Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments](https://mlsanity.com/paper/1705.01693/) (2017)
- [Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles](https://mlsanity.com/paper/1812.06120/) (2018)
- [Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning](https://mlsanity.com/paper/1701.08832/) (2017)
- [Reinforcement Learning for Mixed Autonomy Intersections](https://mlsanity.com/paper/2111.04686/) (2021)
- [Polling-systems-based Autonomous Vehicle Coordination in Traffic Intersections with No Traffic Signals](https://mlsanity.com/paper/1607.07896/) (2016)
- [Optimizing Mixed Autonomy Traffic Flow With Decentralized Autonomous Vehicles and Multi-Agent RL](https://mlsanity.com/paper/2011.00120/) (2020)

## Built on by (most cited first, 13 of 13)

- [Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate](https://mlsanity.com/paper/2210.07553/) (2022)
- [Learning to Control and Coordinate Mixed Traffic Through Robot Vehicles at Complex and Unsignalized Intersections](https://mlsanity.com/paper/2301.05294/) (2023)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning](https://mlsanity.com/paper/2410.15221/) (2024)
- [Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomy](https://mlsanity.com/paper/2312.09436/) (2023)
- [Effective Multi-Agent Deep Reinforcement Learning Control with Relative Entropy Regularization](https://mlsanity.com/paper/2309.14727/) (2023)
- [Multi-agent Path Finding for Cooperative Autonomous Driving](https://mlsanity.com/paper/2402.00334/) (2024)
- [V2X-Lead: LiDAR-based End-to-End Autonomous Driving with Vehicle-to-Everything Communication Integration](https://mlsanity.com/paper/2309.15252/) (2023)
- [Communication-Aware Reinforcement Learning for Cooperative Adaptive Cruise Control](https://mlsanity.com/paper/2407.08964/) (2024)
- [Mitigating Metropolitan Carbon Emissions with Dynamic Eco-driving at Scale](https://mlsanity.com/paper/2408.05609/) (2024)
- [Cooperative Advisory Residual Policies for Congestion Mitigation](https://mlsanity.com/paper/2407.00553/) (2024)
- [Multi-agent Path Finding for Mixed Autonomy Traffic Coordination](https://mlsanity.com/paper/2409.03881/) (2024)
- [Lessons in Cooperation: A Qualitative Analysis of Driver Sentiments towards Real-Time Advisory Systems from a Driving Simulator User Study](https://mlsanity.com/paper/2407.13775/) (2024)

## Abstract (arXiv)

Emerging vehicular systems with increasing proportions of automated components present opportunities for optimal control to mitigate congestion and increase efficiency. There has been a recent interest in applying deep reinforcement learning (DRL) to these nonlinear dynamical systems for the automatic design of effective control strategies. Despite conceptual advantages of DRL being model-free, studies typically nonetheless rely on training setups that are painstakingly specialized to specific vehicular systems. This is a key challenge to efficient analysis of diverse vehicular and mobility systems. To this end, this article contributes a streamlined methodology for vehicular microsimulation and discovers high performance control strategies with minimal manual design. A variable-agent, multi-task approach is presented for optimization of vehicular Partially Observed Markov Decision Processes. The methodology is experimentally validated on mixed autonomy traffic systems, where fractions of vehicles are automated; empirical improvement, typically 15-60% over a human driving baseline, is observed in all configurations of six diverse open or closed traffic systems. The study reveals numerous emergent behaviors resembling wave mitigation, traffic signaling, and ramp metering. Finally, the emergent behaviors are analyzed to produce interpretable control strategies, which are validated against the learned control strategies.