# On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles

arXiv: https://arxiv.org/abs/2505.02274 · HTML: https://mlsanity.com/paper/2505.02274/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 59 references that are in the map.
- First posted on arXiv 4 May 2025.

## Builds on (most cited first, 14 of 14)

- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method](https://mlsanity.com/paper/2003.01197/) (2020)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [The social dilemma of autonomous vehicles](https://mlsanity.com/paper/1510.03346/) (2015)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques](https://mlsanity.com/paper/1605.04965/) (2016)
- [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021)
- [Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing](https://mlsanity.com/paper/1908.06540/) (2019)
- [Assessing Safety-Critical Systems from Operational Testing: A Study on Autonomous Vehicles](https://mlsanity.com/paper/2008.09510/) (2020)
- [Coverage based testing for V&V and Safety Assurance of Self-driving Autonomous Vehicles: A Systematic Literature Review](https://mlsanity.com/paper/2103.04364/) (2021)
- [Reliability Analysis of Artificial Intelligence Systems Using Recurrent Events Data from Autonomous Vehicles](https://mlsanity.com/paper/2102.01740/) (2021)
- [The Unnecessity of Assuming Statistically Independent Tests in Bayesian Software Reliability Assessments](https://mlsanity.com/paper/2208.00462/) (2022)
- [Planning Reliability Assurance Tests for Autonomous Vehicles](https://mlsanity.com/paper/2312.00186/) (2023)
- [Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration](https://mlsanity.com/paper/2402.07031/) (2024)

## Built on by (most cited first, 4 of 4)

- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [Quantifying Fidelity: A Decisive Feature Approach to Comparing Synthetic and Real Imagery](https://mlsanity.com/paper/2512.16468/) (2025)
- [Bayesian Optimization applied for accelerated Virtual Validation of the Autonomous Driving Function](https://mlsanity.com/paper/2507.22769/) (2025)
- [Real-World Perturbation Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2607.04953/) (2026)

## Abstract (arXiv)

Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (textitpfs) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims.