# Planning on the fast lane: Learning to interact using attention mechanisms in path integral inverse reinforcement learning

arXiv: https://arxiv.org/abs/2007.05798 · HTML: https://mlsanity.com/paper/2007.05798/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Imitation and offline RL for driving, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 11 Jul 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation and offline RL for driving](https://mlsanity.com/offlinerl/imitation-and-offline-rl-for-driving/)

## Builds on (most cited first, 11 of 11)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Driving](https://mlsanity.com/paper/1905.00229/) (2019)
- [HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards](https://mlsanity.com/paper/1604.06508/) (2016)
- [Driving Style Encoder: Situational Reward Adaptation for General-Purpose Planning in Automated Driving](https://mlsanity.com/paper/1912.03509/) (2019)
- [Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling](https://mlsanity.com/paper/1803.00444/) (2018)

## Built on by (most cited first, 4 of 4)

- [DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving](https://mlsanity.com/paper/2310.05885/) (2023)
- [How To Not Drive: Learning Driving Constraints from Demonstration](https://mlsanity.com/paper/2110.00645/) (2021)
- [Calibration of Human Driving Behavior and Preference Using Naturalistic Traffic Data](https://mlsanity.com/paper/2105.01820/) (2021)
- [Pixel State Value Network for Combined Prediction and Planning in Interactive Environments](https://mlsanity.com/paper/2310.07706/) (2023)

## Abstract (arXiv)

General-purpose trajectory planning algorithms for automated driving utilize complex reward functions to perform a combined optimization of strategic, behavioral, and kinematic features. The specification and tuning of a single reward function is a tedious task and does not generalize over a large set of traffic situations. Deep learning approaches based on path integral inverse reinforcement learning have been successfully applied to predict local situation-dependent reward functions using features of a set of sampled driving policies. Sample-based trajectory planning algorithms are able to approximate a spatio-temporal subspace of feasible driving policies that can be used to encode the context of a situation. However, the interaction with dynamic objects requires an extended planning horizon, which depends on sequential context modeling. In this work, we are concerned with the sequential reward prediction over an extended time horizon. We present a neural network architecture that uses a policy attention mechanism to generate a low-dimensional context vector by concentrating on trajectories with a human-like driving style. Apart from this, we propose a temporal attention mechanism to identify context switches and allow for stable adaptation of rewards. We evaluate our results on complex simulated driving situations, including other moving vehicles. Our evaluation shows that our policy attention mechanism learns to focus on collision-free policies in the configuration space. Furthermore, the temporal attention mechanism learns persistent interaction with other vehicles over an extended planning horizon.