# A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation

arXiv: https://arxiv.org/abs/2110.14870 · HTML: https://mlsanity.com/paper/2110.14870/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [Scenic: A Language for Scenario Specification and Scene Generation](https://mlsanity.com/paper/1809.09310/) (2018)
- [What-If Motion Prediction for Autonomous Driving](https://mlsanity.com/paper/2008.10587/) (2020)
- [Scenic: A Language for Scenario Specification and Data Generation](https://mlsanity.com/paper/2010.06580/) (2020)
- [Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World](https://mlsanity.com/paper/2003.07739/) (2020)
- [Hierarchical Recurrent Attention Networks for Structured Online Maps](https://mlsanity.com/paper/2012.12314/) (2020)
- [Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI](https://mlsanity.com/paper/2005.07173/) (2020)
- [Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction](https://mlsanity.com/paper/2008.06020/) (2020)
- [A Programmatic and Semantic Approach to Explaining and DebuggingNeural Network Based Object Detectors](https://mlsanity.com/paper/1912.00289/) (2019)
- [Parallel and Multi-Objective Falsification with Scenic and VerifAI](https://mlsanity.com/paper/2107.04164/) (2021)
- [Addressing the IEEE AV Test Challenge with Scenic and VerifAI](https://mlsanity.com/paper/2108.13796/) (2021)

## Built on by (most cited first, 6 of 6)

- [Vehicle trajectory prediction works, but not everywhere](https://mlsanity.com/paper/2112.03909/) (2021)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test](https://mlsanity.com/paper/2503.02911/) (2025)
- [Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis](https://mlsanity.com/paper/2211.02736/) (2022)
- [Scenario-based Evaluation of Prediction Models for Automated Vehicles](https://mlsanity.com/paper/2210.06553/) (2022)
- [Bayesian Optimization applied for accelerated Virtual Validation of the Autonomous Driving Function](https://mlsanity.com/paper/2507.22769/) (2025)

## Abstract (arXiv)

Behavior prediction remains one of the most challenging tasks in the autonomous vehicle (AV) software stack. Forecasting the future trajectories of nearby agents plays a critical role in ensuring road safety, as it equips AVs with the necessary information to plan safe routes of travel. However, these prediction models are data-driven and trained on data collected in real life that may not represent the full range of scenarios an AV can encounter. Hence, it is important that these prediction models are extensively tested in various test scenarios involving interactive behaviors prior to deployment. To support this need, we present a simulation-based testing platform which supports (1) intuitive scenario modeling with a probabilistic programming language called Scenic, (2) specifying a multi-objective evaluation metric with a partial priority ordering, (3) falsification of the provided metric, and (4) parallelization of simulations for scalable testing. As a part of the platform, we provide a library of 25 Scenic programs that model challenging test scenarios involving interactive traffic participant behaviors. We demonstrate the effectiveness and the scalability of our platform by testing a trained behavior prediction model and searching for failure scenarios.