mlsanityData to 14 Oct 2026

Statistical Perspectives on Reliability of Artificial Intelligence Systems

arXiv 2111.05391 ↗First posted 9 Nov 2021cs.SE cs.AI stat.APcited by 10

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) systems have become increasingly popular in many areas. Nevertheless, AI technologies are still in their developing stages, and many issues need to be addressed. Among those, the reliability of AI systems needs to be demonstrated so that the AI systems can be used with confidence by the general public. In this paper, we provide statistical perspectives on the reliability of AI systems. Different from other considerations, the reliability of AI systems focuses on the time dimension. That is, the system can perform its designed functionality for the intended period. We introduce a so-called SMART statistical framework for AI reliability research, which includes five components: Structure of the system, Metrics of reliability, Analysis of failure causes, Reliability assessment, and Test planning. We review traditional methods in reliability data analysis and software reliability, and discuss how those existing methods can be transformed for reliability modeling and assessment of AI systems. We also describe recent developments in modeling and analysis of AI reliability and outline statistical research challenges in this area, including out-of-distribution detection, the effect of the training set, adversarial attacks, model accuracy, and uncertainty quantification, and discuss how those topics can be related to AI reliability, with illustrative examples. Finally, we discuss data collection and test planning for AI reliability assessment and how to improve system designs for higher AI reliability. The paper closes with some concluding remarks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 32 of its 110 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

32 of 110 references.

References · 32

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2013Intriguing properties of neural networkscited by 5,155
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  7. 2016Variational Inference: A Review for Statisticianscited by 1,472
  8. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  9. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  10. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  11. 2017Interpretation of Neural Networks is Fragilecited by 444
  12. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
20 more
  1. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  2. 2018Physical Adversarial Examples for Object Detectorscited by 163
  3. 2017EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examplescited by 155
  4. 2019Deep Learning vs. Traditional Computer Visioncited by 94
  5. 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 61
  6. 2020AI4COVID-19: AI Enabled Preliminary Diagnosis for COVID-19 from Cough Samples via an Appcited by 48
  7. 2020Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real Worldcited by 39
  8. 2019The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort studycited by 38
  9. 2016Variational Boosting: Iteratively Refining Posterior Approximationscited by 37
  10. 2019Robustness Verification of Tree-based Modelscited by 33
  11. 2019Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matricescited by 32
  12. 2020A Safety Framework for Critical Systems Utilising Deep Neural Networkscited by 19
  13. 2019Assessing the Safety and Reliability of Autonomous Vehicles from Road Testingcited by 15
  14. 2018Failure Prediction for Autonomous Drivingcited by 14
  15. 2018Variational Rejection Samplingcited by 13
  16. 2021Reliability Analysis of Artificial Intelligence Systems Using Recurrent Events Data from Autonomous Vehiclescited by 6
  17. 2019Challenges of Reliability Assessment and Enhancement in Autonomous Systemscited by 3
  18. 2018Software Reliability Growth Models Predict Autonomous Vehicle Disengagement Eventscited by 2
  19. 2021Special Session: Reliability Analysis for ML/AI Hardwarecited by 2
  20. 2019A Case for Quantifying Statistical Robustness of Specialized Probabilistic AI Acceleratorscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 3.

10 papers.

  1. 2025StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysiscited by 8
  2. 2024Trustworthy and Responsible AI for Human-Centric Autonomous Decision-Making Systemscited by 3
  3. 2023Planning Reliability Assurance Tests for Autonomous Vehiclescited by 2
  4. 2024ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNscited by 1
  5. 2024A Comprehensive Case Study on the Performance of Machine Learning Methods on the Classification of Solar Panel Electroluminescence Imagescited by 1
  6. 2024Contextualized AI for Cyber Defense: An Automated Survey using LLMscited by 1
  7. 2025Bridging the Data Gap in AI Reliability Research and Establishing DR-AIR, a Comprehensive Data Repository for AI Reliabilitycited by 1
  8. 2022Artificial Intelligence Nomenclature Identified From Delphi Study on Key Issues Related to Trust and Barriers to Adoption for Autonomous Systemscited by 0
  9. 2023Deep Neural Network Identification of Limnonectes Species and New Class Detection Using Image Datacited by 0
  10. 2025Detecting underdetermination in parameterized quantum circuitscited by 0