# Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation

arXiv: https://arxiv.org/abs/2505.06134 · HTML: https://mlsanity.com/paper/2505.06134/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 17 of 17)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2201.05057/) (2022)
- [FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs](https://mlsanity.com/paper/2211.16197/) (2022)
- [ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation](https://mlsanity.com/paper/2307.14187/) (2023)
- [Are socially-aware trajectory prediction models really socially-aware?](https://mlsanity.com/paper/2108.10879/) (2021)
- [Training Meta-Surrogate Model for Transferable Adversarial Attack](https://mlsanity.com/paper/2109.01983/) (2021)
- [Semi-supervised Semantics-guided Adversarial Training for Trajectory Prediction](https://mlsanity.com/paper/2205.14230/) (2022)
- [Targeted Adversarial Attacks against Neural Network Trajectory Predictors](https://mlsanity.com/paper/2212.04138/) (2022)
- [A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities](https://mlsanity.com/paper/2210.08906/) (2022)
- [Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios](https://mlsanity.com/paper/2211.05455/) (2022)
- [Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness](https://mlsanity.com/paper/2403.14641/) (2024)
- [SA-Attack: Speed-adaptive stealthy adversarial attack on trajectory prediction](https://mlsanity.com/paper/2404.12612/) (2024)

## Built on by (most cited first, 3 of 3)

- [STEP: Structured Training and Evaluation Platform for benchmarking trajectory prediction models](https://mlsanity.com/paper/2509.14801/) (2025)
- [Evaluating randomized smoothing as a defense against adversarial attacks in trajectory prediction](https://mlsanity.com/paper/2603.10821/) (2026)
- [Clustered Randomized Smoothing for Stochastic Prediction Functions](https://mlsanity.com/paper/2608.12037/) (2026)

## Abstract (arXiv)

Trajectory prediction is a key element of autonomous vehicle systems, enabling them to anticipate and react to the movements of other road users. Evaluating the robustness of prediction models against adversarial attacks is essential to ensure their reliability in real-world traffic. However, current approaches tend to focus on perturbing the past positions of surrounding agents, which can generate unrealistic scenarios and overlook critical vulnerabilities. This limitation may result in overly optimistic assessments of model performance in real-world conditions. In this work, we demonstrate that perturbing not just past but also future states of adversarial agents can uncover previously undetected weaknesses and thereby provide a more rigorous evaluation of model robustness. Our novel approach incorporates dynamic constraints and preserves tactical behaviors, enabling more effective and realistic adversarial attacks. We introduce new performance measures to assess the realism and impact of these adversarial trajectories. Testing our method on a state-of-the-art prediction model revealed significant increases in prediction errors and collision rates under adversarial conditions. Qualitative analysis further showed that our attacks can expose critical weaknesses, such as the inability of the model to detect potential collisions in what appear to be safe predictions. These results underscore the need for more comprehensive adversarial testing to better evaluate and improve the reliability of trajectory prediction models for autonomous vehicles.