# Demystifying the Physics of Deep Reinforcement Learning-Based Autonomous Vehicle Decision-Making

arXiv: https://arxiv.org/abs/2403.11432 · HTML: https://mlsanity.com/paper/2403.11432/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention](https://mlsanity.com/paper/1703.10631/) (2017)
- [A Survey of Deep Learning Applications to Autonomous Vehicle Control](https://mlsanity.com/paper/1912.10773/) (2019)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1901.00569/) (2019)
- [Rethinking Attention-Model Explainability through Faithfulness Violation Test](https://mlsanity.com/paper/2201.12114/) (2022)
- [Social Attention for Autonomous Decision-Making in Dense Traffic](https://mlsanity.com/paper/1911.12250/) (2019)
- [Game-Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems](https://mlsanity.com/paper/1608.08589/) (2016)
- [Deep Reinforcement Learning with Enhanced Safety for Autonomous Highway Driving](https://mlsanity.com/paper/1910.12905/) (2019)
- [Iterative Causal Discovery in the Possible Presence of Latent Confounders and Selection Bias](https://mlsanity.com/paper/2111.04095/) (2021)
- [Decision-Making Technology for Autonomous Vehicles Learning-Based Methods, Applications and Future Outlook](https://mlsanity.com/paper/2107.01110/) (2021)
- [From Temporal to Contemporaneous Iterative Causal Discovery in the Presence of Latent Confounders](https://mlsanity.com/paper/2306.00624/) (2023)
- [Predicting optimal value functions by interpolating reward functions in scalarized multi-objective reinforcement learning](https://mlsanity.com/paper/1909.05004/) (2019)

## Abstract (arXiv)

With the advent of universal function approximators in the domain of reinforcement learning, the number of practical applications leveraging deep reinforcement learning (DRL) has exploded. Decision-making in autonomous vehicles (AVs) has emerged as a chief application among them, taking the sensor data or the higher-order kinematic variables as the input and providing a discrete choice or continuous control output. There has been a continuous effort to understand the black-box nature of the DRL models, but so far, there hasn't been any discussion (to the best of authors' knowledge) about how the models learn the physical process. This presents an overwhelming limitation that restricts the real-world deployment of DRL in AVs. Therefore, in this research work, we try to decode the knowledge learnt by the attention-based DRL framework about the physical process. We use a continuous proximal policy optimization-based DRL algorithm as the baseline model and add a multi-head attention framework in an open-source AV simulation environment. We provide some analytical techniques for discussing the interpretability of the trained models in terms of explainability and causality for spatial and temporal correlations. We show that the weights in the first head encode the positions of the neighboring vehicles while the second head focuses on the leader vehicle exclusively. Also, the ego vehicle's action is causally dependent on the vehicles in the target lane spatially and temporally. Through these findings, we reliably show that these techniques can help practitioners decipher the results of the DRL algorithms.