# Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing

arXiv: https://arxiv.org/abs/1908.06540 · HTML: https://mlsanity.com/paper/1908.06540/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 15 of 15)

- [Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance](https://mlsanity.com/paper/2112.00646/) (2021)
- [Statistical Perspectives on Reliability of Artificial Intelligence Systems](https://mlsanity.com/paper/2111.05391/) (2021)
- [Reliability Analysis of Artificial Intelligence Systems Using Recurrent Events Data from Autonomous Vehicles](https://mlsanity.com/paper/2102.01740/) (2021)
- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles](https://mlsanity.com/paper/2505.02274/) (2025)
- [Safety Case Templates for Autonomous Systems](https://mlsanity.com/paper/2102.02625/) (2021)
- [Assessing Confidence with Assurance 2.0](https://mlsanity.com/paper/2205.04522/) (2022)
- [Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems](https://mlsanity.com/paper/2210.14991/) (2022)
- [Symbiotic System of Systems Design for Safe and Resilient Autonomous Robotics in Offshore Wind Farms](https://mlsanity.com/paper/2101.09491/) (2021)
- [Bootstrapping confidence in future safety based on past safe operation](https://mlsanity.com/paper/2110.10718/) (2021)
- [Planning Reliability Assurance Tests for Autonomous Vehicles](https://mlsanity.com/paper/2312.00186/) (2023)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [Generating Critical Scenarios for Testing Automated Driving Systems](https://mlsanity.com/paper/2412.02574/) (2024)
- [Towards provable probabilistic safety for scalable embodied AI systems](https://mlsanity.com/paper/2506.05171/) (2025)
- [A Computationally Efficient Learning of Artificial Intelligence System Reliability Considering Error Propagation](https://mlsanity.com/paper/2603.18201/) (2026)

## Abstract (arXiv)

There is an urgent societal need to assess whether autonomous vehicles (AVs) are safe enough. From published quantitative safety and reliability assessments of AVs, we know that, given the goal of predicting very low rates of accidents, road testing alone requires infeasible numbers of miles to be driven. However, previous analyses do not consider any knowledge prior to road testing - knowledge which could bring substantial advantages if the AV design allows strong expectations of safety before road testing. We present the advantages of a new variant of Conservative Bayesian Inference (CBI), which uses prior knowledge while avoiding optimistic biases. We then study the trend of disengagements (take-overs by human drivers) by applying Software Reliability Growth Models (SRGMs) to data from Waymo's public road testing over 51 months, in view of the practice of software updates during this testing. Our approach is to not trust any specific SRGM, but to assess forecast accuracy and then improve forecasts. We show that, coupled with accuracy assessment and recalibration techniques, SRGMs could be a valuable test planning aid.