# Reliability Analysis of Artificial Intelligence Systems Using Recurrent Events Data from Autonomous Vehicles

arXiv: https://arxiv.org/abs/2102.01740 · HTML: https://mlsanity.com/paper/2102.01740/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 48 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 8 of 8)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Smoothing parameter and model selection for general smooth models](https://mlsanity.com/paper/1511.03864/) (2015)
- [A Safety Framework for Critical Systems Utilising Deep Neural Networks](https://mlsanity.com/paper/2003.05311/) (2020)
- [Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing](https://mlsanity.com/paper/1908.06540/) (2019)
- [Software Reliability Growth Models Predict Autonomous Vehicle Disengagement Events](https://mlsanity.com/paper/1812.08901/) (2018)
- [Semi-parametric Models for Accelerated Destructive Degradation Test Data Analysis](https://mlsanity.com/paper/1512.03036/) (2015)
- [A Flexible Parametric Modelling Framework for Survival Analysis](https://mlsanity.com/paper/1901.03212/) (2019)

## Built on by (most cited first, 6 of 6)

- [Statistical Perspectives on Reliability of Artificial Intelligence Systems](https://mlsanity.com/paper/2111.05391/) (2021)
- [On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles](https://mlsanity.com/paper/2505.02274/) (2025)
- [Planning Reliability Assurance Tests for Autonomous Vehicles](https://mlsanity.com/paper/2312.00186/) (2023)
- [Contextualized AI for Cyber Defense: An Automated Survey using LLMs](https://mlsanity.com/paper/2409.13524/) (2024)
- [Bridging the Data Gap in AI Reliability Research and Establishing DR-AIR, a Comprehensive Data Repository for AI Reliability](https://mlsanity.com/paper/2502.12386/) (2025)
- [A Computationally Efficient Learning of Artificial Intelligence System Reliability Considering Error Propagation](https://mlsanity.com/paper/2603.18201/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing, and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modeling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modeling and analyzing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new nonparametric model based on monotonic splines to describe the event process. We develop inference procedures for selecting the best models, quantifying uncertainty, and testing heterogeneity in the event process. We then analyze the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems.