# PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems

arXiv: https://arxiv.org/abs/2308.00864 · HTML: https://mlsanity.com/paper/2308.00864/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 26 of 26)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis](https://mlsanity.com/paper/1807.07814/) (2018)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [Flow: A Modular Learning Framework for Mixed Autonomy Traffic](https://mlsanity.com/paper/1710.05465/) (2017)
- [How would surround vehicles move? A Unified Framework for Maneuver Classification and Motion Prediction](https://mlsanity.com/paper/1801.06523/) (2018)
- [Dissipation of stop-and-go waves via control of autonomous vehicles: Field experiments](https://mlsanity.com/paper/1705.01693/) (2017)
- [Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationships](https://mlsanity.com/paper/2011.04251/) (2020)
- [Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing](https://mlsanity.com/paper/1808.03246/) (2018)
- [Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning](https://mlsanity.com/paper/2002.06836/) (2020)
- [Residual Policy Learning for Shared Autonomy](https://mlsanity.com/paper/2004.05097/) (2020)
- [Influencing Towards Stable Multi-Agent Interactions](https://mlsanity.com/paper/2110.08229/) (2021)
- [Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics](https://mlsanity.com/paper/2211.02231/) (2022)
- [Simultaneous Policy Learning and Latent State Inference for Imitating Driver Behavior](https://mlsanity.com/paper/1704.05566/) (2017)
- [Reinforcement Learning for Mixed Autonomy Intersections](https://mlsanity.com/paper/2111.04686/) (2021)
- [Long-term Pedestrian Trajectory Prediction using Mutable Intention Filter and Warp LSTM](https://mlsanity.com/paper/2007.00113/) (2020)
- [A Taxonomy and Review of Algorithms for Modeling and Predicting Human Driver Behavior](https://mlsanity.com/paper/2006.08832/) (2020)
- [Learning to Navigate Intersections with Unsupervised Driver Trait Inference](https://mlsanity.com/paper/2109.06783/) (2021)
- [RILI: Robustly Influencing Latent Intent](https://mlsanity.com/paper/2203.12705/) (2022)
- [Data-Augmented Contact Model for Rigid Body Simulation](https://mlsanity.com/paper/1803.04019/) (2018)
- [Meta-path Analysis on Spatio-Temporal Graphs for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2202.13427/) (2022)
- [Towards Co-operative Congestion Mitigation](https://mlsanity.com/paper/2302.09140/) (2023)

## Built on by (most cited first, 4 of 4)

- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [Temporal Transfer Learning for Traffic Optimization with Coarse-grained Advisory Autonomy](https://mlsanity.com/paper/2312.09436/) (2023)
- [Cooperative Advisory Residual Policies for Congestion Mitigation](https://mlsanity.com/paper/2407.00553/) (2024)
- [Lessons in Cooperation: A Qualitative Analysis of Driver Sentiments towards Real-Time Advisory Systems from a Driving Simulator User Study](https://mlsanity.com/paper/2407.13775/) (2024)

## Abstract (arXiv)

Intelligent driving systems can be used to mitigate congestion through simple actions, thus improving many socioeconomic factors such as commute time and gas costs. However, these systems assume precise control over autonomous vehicle fleets, and are hence limited in practice as they fail to account for uncertainty in human behavior. Piecewise Constant (PC) Policies address these issues by structurally modeling the likeness of human driving to reduce traffic congestion in dense scenarios to provide action advice to be followed by human drivers. However, PC policies assume that all drivers behave similarly. To this end, we develop a co-operative advisory system based on PC policies with a novel driver trait conditioned Personalized Residual Policy, PeRP. PeRP advises drivers to behave in ways that mitigate traffic congestion. We first infer the driver's intrinsic traits on how they follow instructions in an unsupervised manner with a variational autoencoder. Then, a policy conditioned on the inferred trait adapts the action of the PC policy to provide the driver with a personalized recommendation. Our system is trained in simulation with novel driver modeling of instruction adherence. We show that our approach successfully mitigates congestion while adapting to different driver behaviors, with 4 to 22% improvement in average speed over baselines.