# Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomy

arXiv: https://arxiv.org/abs/2312.09436 · HTML: https://mlsanity.com/paper/2312.09436/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [On the Theory of Transfer Learning: The Importance of Task Diversity](https://mlsanity.com/paper/2006.11650/) (2020)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [An introduction to domain adaptation and transfer learning](https://mlsanity.com/paper/1812.11806/) (2018)
- [Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis](https://mlsanity.com/paper/1807.07814/) (2018)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Flow: A Modular Learning Framework for Mixed Autonomy Traffic](https://mlsanity.com/paper/1710.05465/) (2017)
- [Contextualize Me -- The Case for Context in Reinforcement Learning](https://mlsanity.com/paper/2202.04500/) (2022)
- [Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments](https://mlsanity.com/paper/1705.01693/) (2017)
- [Learning What and Where to Transfer](https://mlsanity.com/paper/1905.05901/) (2019)
- [Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning](https://mlsanity.com/paper/1702.06054/) (2017)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [On the Generalization Gap in Reparameterizable Reinforcement Learning](https://mlsanity.com/paper/1905.12654/) (2019)
- [Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles](https://mlsanity.com/paper/1812.06120/) (2018)
- [Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning](https://mlsanity.com/paper/1701.08832/) (2017)
- [Context-Aware Policy Reuse](https://mlsanity.com/paper/1806.03793/) (2018)
- [Reinforcement Learning for Mixed Autonomy Intersections](https://mlsanity.com/paper/2111.04686/) (2021)
- [Unified Automatic Control of Vehicular Systems with Reinforcement Learning](https://mlsanity.com/paper/2208.00268/) (2022)
- [The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning](https://mlsanity.com/paper/2210.08607/) (2022)
- [Inter-Level Cooperation in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1912.02368/) (2019)
- [Multi-Robot Transfer Learning: A Dynamical System Perspective](https://mlsanity.com/paper/1707.08689/) (2017)
- [Coarse-to-fine: A RNN-based hierarchical attention model for vehicle re-identification](https://mlsanity.com/paper/1812.04239/) (2018)
- [Transfer learning to improve streamflow forecasts in data sparse regions](https://mlsanity.com/paper/2112.03088/) (2021)
- [PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems](https://mlsanity.com/paper/2308.00864/) (2023)
- [Towards Co-operative Congestion Mitigation](https://mlsanity.com/paper/2302.09140/) (2023)
- [Integrated Analysis of Coarse-Grained Guidance for Traffic Flow Stability](https://mlsanity.com/paper/2301.04043/) (2023)

## Built on by (most cited first, 6 of 6)

- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [Structure Detection for Contextual Reinforcement Learning](https://mlsanity.com/paper/2601.08120/) (2026)
- [Cooperative Advisory Residual Policies for Congestion Mitigation](https://mlsanity.com/paper/2407.00553/) (2024)
- [Lessons in Cooperation: A Qualitative Analysis of Driver Sentiments towards Real-Time Advisory Systems from a Driving Simulator User Study](https://mlsanity.com/paper/2407.13775/) (2024)
- [Formalizing Task-Space Complexity for Zero-Shot Generalization](https://mlsanity.com/paper/2606.20967/) (2026)
- [Task Specialization Fine-Tuning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2608.17180/) (2026)

## Abstract (arXiv)

The recent development of connected and automated vehicle (CAV) technologies has spurred investigations to optimize dense urban traffic to maximize vehicle speed and throughput. This paper explores advisory autonomy, in which real-time driving advisories are issued to the human drivers, thus achieving near-term performance of automated vehicles. Due to the complexity of traffic systems, recent studies of coordinating CAVs have resorted to leveraging deep reinforcement learning (RL). Coarse-grained advisory is formalized as zero-order holds, and we consider a range of hold duration from 0.1 to 40 seconds. However, despite the similarity of the higher frequency tasks on CAVs, a direct application of deep RL fails to be generalized to advisory autonomy tasks. To overcome this, we utilize zero-shot transfer, training policies on a set of source tasks--specific traffic scenarios with designated hold durations--and then evaluating the efficacy of these policies on different target tasks. We introduce Temporal Transfer Learning (TTL) algorithms to select source tasks for zero-shot transfer, systematically leveraging the temporal structure to solve the full range of tasks. TTL selects the most suitable source tasks to maximize the performance of the range of tasks. We validate our algorithms on diverse mixed-traffic scenarios, demonstrating that TTL more reliably solves the tasks than baselines. This paper underscores the potential of coarse-grained advisory autonomy with TTL in traffic flow optimization.