# Real-time system optimal traffic routing under uncertainties -- Can physics models boost reinforcement learning?

arXiv: https://arxiv.org/abs/2407.07364 · HTML: https://mlsanity.com/paper/2407.07364/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 71 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey](https://mlsanity.com/paper/2005.00935/) (2020)
- [Measuring the Reliability of Reinforcement Learning Algorithms](https://mlsanity.com/paper/1912.05663/) (2019)
- [Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1901.00569/) (2019)
- [Bridging the Imitation Gap by Adaptive Insubordination](https://mlsanity.com/paper/2007.12173/) (2020)
- [Prototypical Reward Network for Data-Efficient RLHF](https://mlsanity.com/paper/2406.06606/) (2024)
- [TGRL: An Algorithm for Teacher Guided Reinforcement Learning](https://mlsanity.com/paper/2307.03186/) (2023)
- [Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment](https://mlsanity.com/paper/1804.06264/) (2018)
- [Hybrid Reinforcement Learning-Based Eco-Driving Strategy for Connected and Automated Vehicles at Signalized Intersections](https://mlsanity.com/paper/2201.07833/) (2022)
- [Learning How to Dynamically Route Autonomous Vehicles on Shared Roads](https://mlsanity.com/paper/1909.03664/) (2019)
- [Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning](https://mlsanity.com/paper/2104.09936/) (2021)
- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [Estimating multi-class dynamic origin-destination demand through a forward-backward algorithm on computational graphs](https://mlsanity.com/paper/1903.04681/) (2019)
- [Deep Reinforcement Learning Algorithm for Dynamic Pricing of Express Lanes with Multiple Access Locations](https://mlsanity.com/paper/1909.04760/) (2019)

## Built on by (most cited first, 7 of 7)

- [Interpretable mixture of experts for time series prediction under recurrent and non-recurrent conditions](https://mlsanity.com/paper/2409.03282/) (2024)
- [UniLCD: Unified Local-Cloud Decision-Making via Reinforcement Learning](https://mlsanity.com/paper/2409.11403/) (2024)
- [Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models](https://mlsanity.com/paper/2410.19241/) (2024)
- [Reinforcement Learning-based Sequential Route Recommendation for System-Optimal Traffic Assignment](https://mlsanity.com/paper/2505.20889/) (2025)
- [SafeAug: Safety-Critical Driving Data Augmentation from Naturalistic Datasets](https://mlsanity.com/paper/2501.02143/) (2025)
- [Know Unreported Roadway Incidents in Real-time: Early Traffic Anomaly Detection](https://mlsanity.com/paper/2412.10892/) (2024)
- [Scalable Dynamic Origin-Destination Demand Estimation Enhanced by High-Resolution Satellite Imagery Data](https://mlsanity.com/paper/2506.22499/) (2025)

## Abstract (arXiv)

System optimal traffic routing can mitigate congestion by assigning routes for a portion of vehicles so that the total travel time of all vehicles in the transportation system can be reduced. However, achieving real-time optimal routing poses challenges due to uncertain demands and unknown system dynamics, particularly in expansive transportation networks. While physics model-based methods are sensitive to uncertainties and model mismatches, model-free reinforcement learning struggles with learning inefficiencies and interpretability issues. Our paper presents TransRL, a novel algorithm that integrates reinforcement learning with physics models for enhanced performance, reliability, and interpretability. TransRL begins by establishing a deterministic policy grounded in physics models, from which it learns from and is guided by a differentiable and stochastic teacher policy. During training, TransRL aims to maximize cumulative rewards while minimizing the Kullback Leibler (KL) divergence between the current policy and the teacher policy. This approach enables TransRL to simultaneously leverage interactions with the environment and insights from physics models. We conduct experiments on three transportation networks with up to hundreds of links. The results demonstrate TransRL's superiority over traffic model-based methods for being adaptive and learning from the actual network data. By leveraging the information from physics models, TransRL consistently outperforms state-of-the-art reinforcement learning algorithms such as proximal policy optimization (PPO) and soft actor critic (SAC). Moreover, TransRL's actions exhibit higher reliability and interpretability compared to baseline reinforcement learning approaches like PPO and SAC.