# Proximal Policy Optimization Learning based Control of Congested Freeway Traffic

arXiv: https://arxiv.org/abs/2204.05627 · HTML: https://mlsanity.com/paper/2204.05627/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Truly Proximal Policy Optimization](https://mlsanity.com/paper/1903.07940/) (2019)
- [Smoothing Traffic Flow via Control of Autonomous Vehicles](https://mlsanity.com/paper/1812.09544/) (2018)
- [Delay-robust control design for two heterodirectional linear coupled hyperbolic PDEs](https://mlsanity.com/paper/1709.04274/) (2017)
- [Delay-Compensated Distributed PDE Control of Traffic with Connected/Automated Vehicles](https://mlsanity.com/paper/2107.08651/) (2021)
- [PDE-Based Feedback Control of Freeway Traffic Flow via Time-Gap Manipulation of ACC-Equipped Vehicles](https://mlsanity.com/paper/1812.08154/) (2018)
- [Output Feedback Control of Two-lane Traffic Congestion](https://mlsanity.com/paper/1905.01571/) (2019)

## Built on by (most cited first, 1 of 1)

- [Neural Operator based Reinforcement Learning for Control of first-order PDEs with Spatially-Varying State Delay](https://mlsanity.com/paper/2501.18201/) (2025)

## Abstract (arXiv)

This study proposes a delay-compensated feedback controller based on proximal policy optimization (PPO) reinforcement learning to stabilize traffic flow in the congested regime by manipulating the time-gap of adaptive cruise control-equipped (ACC-equipped) vehicles.The traffic dynamics on a freeway segment are governed by an Aw-Rascle-Zhang (ARZ) model, consisting of 2× 2 nonlinear first-order partial differential equations (PDEs).Inspired by the backstepping delay compensator [18] but different from whose complex segmented control scheme, the PPO control is composed of three feedbacks, namely the current traffic flow velocity, the current traffic flow density and previous one step control input. The control gains for the three feedbacks are learned from the interaction between the PPO and the numerical simulator of the traffic system without knowing the system dynamics. Numerical simulation experiments are designed to compare the Lyapunov control, the backstepping control and the PPO control. The results show that for a delay-free system, the PPO control has faster convergence rate and less control effort than the Lyapunov control. For a traffic system with input delay, the performance of the PPO controller is comparable to that of the Backstepping controller, even for the situation that the delay value does not match. However, the PPO is robust to parameter perturbations, while the Backstepping controller cannot stabilize a system where one of the parameters is disturbed by Gaussian noise.