# Dynamic Interaction-Aware Scene Understanding for Reinforcement Learning in Autonomous Driving

arXiv: https://arxiv.org/abs/1909.13582 · HTML: https://mlsanity.com/paper/1909.13582/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 14 of 14)

- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [PyG 2.0: Scalable Learning on Real World Graphs](https://mlsanity.com/paper/2507.16991/) (2025)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [Cooperative Behavior Planning for Automated Driving using Graph Neural Networks](https://mlsanity.com/paper/2202.11376/) (2022)
- [Optimizing Trajectories for Highway Driving with Offline Reinforcement Learning](https://mlsanity.com/paper/2203.10949/) (2022)
- [Automatic Intersection Management in Mixed Traffic Using Reinforcement Learning and Graph Neural Networks](https://mlsanity.com/paper/2301.12717/) (2023)
- [Geometric Deep Learning for Autonomous Driving: Unlocking the Power of Graph Neural Networks With CommonRoad-Geometric](https://mlsanity.com/paper/2302.01259/) (2023)
- [Autonomous Navigation through intersections with Graph ConvolutionalNetworks and Conditional Imitation Learning for Self-driving Cars](https://mlsanity.com/paper/2102.00675/) (2021)
- [Deep Occupancy-Predictive Representations for Autonomous Driving](https://mlsanity.com/paper/2303.04218/) (2023)
- [Interaction-Aware Decision-Making for Autonomous Vehicles in Forced Merging Scenario Leveraging Social Psychology Factors](https://mlsanity.com/paper/2309.14497/) (2023)
- [Decision-Making for Autonomous Vehicles with Interaction-Aware Behavioral Prediction and Social-Attention Neural Network](https://mlsanity.com/paper/2310.20148/) (2023)
- [Learning Isometric Embeddings of Road Networks using Multidimensional Scaling](https://mlsanity.com/paper/2504.17534/) (2025)
- [High-Order Evolving Graphs for Enhanced Representation of Traffic Dynamics](https://mlsanity.com/paper/2409.11206/) (2024)
- [Multitask Reinforcement Learning for Assisting Choice Model Specification](https://mlsanity.com/paper/2609.18441/) (2026)

## Abstract (arXiv)

The common pipeline in autonomous driving systems is highly modular and includes a perception component which extracts lists of surrounding objects and passes these lists to a high-level decision component. In this case, leveraging the benefits of deep reinforcement learning for high-level decision making requires special architectures to deal with multiple variable-length sequences of different object types, such as vehicles, lanes or traffic signs. At the same time, the architecture has to be able to cover interactions between traffic participants in order to find the optimal action to be taken. In this work, we propose the novel Deep Scenes architecture, that can learn complex interaction-aware scene representations based on extensions of either 1) Deep Sets or 2) Graph Convolutional Networks. We present the Graph-Q and DeepScene-Q off-policy reinforcement learning algorithms, both outperforming state-of-the-art methods in evaluations with the publicly available traffic simulator SUMO.