# Hacking Predictors Means Hacking Cars: Using Sensitivity Analysis to Identify Trajectory Prediction Vulnerabilities for Autonomous Driving Security

arXiv: https://arxiv.org/abs/2401.10313 · HTML: https://mlsanity.com/paper/2401.10313/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers](https://mlsanity.com/paper/1604.00825/) (2016)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1510.03820/) (2015)
- [On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2201.05057/) (2022)
- [AdvDO: Realistic Adversarial Attacks for Trajectory Prediction](https://mlsanity.com/paper/2209.08744/) (2022)
- [You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction](https://mlsanity.com/paper/2110.05304/) (2021)
- [Robust Trajectory Prediction against Adversarial Attacks](https://mlsanity.com/paper/2208.00094/) (2022)
- [Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachability](https://mlsanity.com/paper/2101.05916/) (2021)
- [Targeted Adversarial Attacks against Neural Network Trajectory Predictors](https://mlsanity.com/paper/2212.04138/) (2022)
- [Multi-Adversarial Safety Analysis for Autonomous Vehicles](https://mlsanity.com/paper/2112.14344/) (2021)
- [Reachability Analysis for FollowerStopper: Safety Analysis and Experimental Results](https://mlsanity.com/paper/2112.14345/) (2021)

## Abstract (arXiv)

Adversarial attacks on learning-based multi-modal trajectory predictors have already been demonstrated. However, there are still open questions about the effects of perturbations on inputs other than state histories, and how these attacks impact downstream planning and control. In this paper, we conduct a sensitivity analysis on two trajectory prediction models, Trajectron++ and AgentFormer. The analysis reveals that between all inputs, almost all of the perturbation sensitivities for both models lie only within the most recent position and velocity states. We additionally demonstrate that, despite dominant sensitivity on state history perturbations, an undetectable image map perturbation made with the Fast Gradient Sign Method can induce large prediction error increases in both models, revealing that these trajectory predictors are, in fact, susceptible to image-based attacks. Using an optimization-based planner and example perturbations crafted from sensitivity results, we show how these attacks can cause a vehicle to come to a sudden stop from moderate driving speeds.