# Scenario-based Evaluation of Prediction Models for Automated Vehicles

arXiv: https://arxiv.org/abs/2210.06553 · HTML: https://mlsanity.com/paper/2210.06553/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [MotionCNN: A Strong Baseline for Motion Prediction in Autonomous Driving](https://mlsanity.com/paper/2206.02163/) (2022)
- [Deep Learning for Vision-based Prediction: A Survey](https://mlsanity.com/paper/2007.00095/) (2020)
- [RobustTP: End-to-End Trajectory Prediction for Heterogeneous Road-Agents in Dense Traffic with Noisy Sensor Inputs](https://mlsanity.com/paper/1907.08752/) (2019)
- [A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation](https://mlsanity.com/paper/2110.14870/) (2021)
- [Towards an Ontology for Scenario Definition for the Assessment of Automated Vehicles: An Object-Oriented Framework](https://mlsanity.com/paper/2001.11507/) (2020)

## Built on by (most cited first, 4 of 4)

- [MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes](https://mlsanity.com/paper/2403.00353/) (2024)
- [Prediction Horizon Requirements for Automated Driving: Optimizing Safety, Comfort, and Efficiency](https://mlsanity.com/paper/2402.03893/) (2024)
- [Robustness Benchmark of Road User Trajectory Prediction Models for Automated Driving](https://mlsanity.com/paper/2304.01895/) (2023)
- [Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving](https://mlsanity.com/paper/2512.12211/) (2025)

## Abstract (arXiv)

To operate safely, an automated vehicle (AV) must anticipate how the environment around it will evolve. For that purpose, it is important to know which prediction models are most appropriate for every situation. Currently, assessment of prediction models is often performed over a set of trajectories without distinction of the type of movement they capture, resulting in the inability to determine the suitability of each model for different situations. In this work we illustrate how standardized evaluation methods result in wrong conclusions regarding a model's predictive capabilities, preventing a clear assessment of prediction models and potentially leading to dangerous on-road situations. We argue that following evaluation practices in safety assessment for AVs, assessment of prediction models should be performed in a scenario-based fashion. To encourage scenario-based assessment of prediction models and illustrate the dangers of improper assessment, we categorize trajectories of the Waymo Open Motion dataset according to the type of movement they capture. Next, three different models are thoroughly evaluated for different trajectory types and prediction horizons. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.