# Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance

arXiv: https://arxiv.org/abs/2112.00646 · HTML: https://mlsanity.com/paper/2112.00646/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 97 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 30 of 30)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Recent Advances in Adversarial Training for Adversarial Robustness](https://mlsanity.com/paper/2102.01356/) (2021)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach](https://mlsanity.com/paper/1801.10578/) (2018)
- [A Tutorial on Kernel Density Estimation and Recent Advances](https://mlsanity.com/paper/1704.03924/) (2017)
- [Analysis of classifiers' robustness to adversarial perturbations](https://mlsanity.com/paper/1502.02590/) (2015)
- [A Statistical Approach to Assessing Neural Network Robustness](https://mlsanity.com/paper/1811.07209/) (2018)
- [Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges](https://mlsanity.com/paper/1905.04223/) (2019)
- [Boosting Operational DNN Testing Efficiency through Conditioning](https://mlsanity.com/paper/1906.02533/) (2019)
- [Distribution-Aware Testing of Neural Networks Using Generative Models](https://mlsanity.com/paper/2102.13602/) (2021)
- [A Safety Framework for Critical Systems Utilising Deep Neural Networks](https://mlsanity.com/paper/2003.05311/) (2020)
- [Statistically Robust Neural Network Classification](https://mlsanity.com/paper/1912.04884/) (2019)
- [Assessing the Safety and Reliability of Autonomous Vehicles from Road Testing](https://mlsanity.com/paper/1908.06540/) (2019)
- [Coverage Guided Testing for Recurrent Neural Networks](https://mlsanity.com/paper/1911.01952/) (2019)
- [Engineering Trustworthy Self-Adaptive Software with Dynamic Assurance Cases](https://mlsanity.com/paper/1703.06350/) (2017)
- [Probabilistic Model Checking of Robots Deployed in Extreme Environments](https://mlsanity.com/paper/1812.04128/) (2018)
- [Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles](https://mlsanity.com/paper/2106.01258/) (2021)
- [Assessing Safety-Critical Systems from Operational Testing: A Study on Autonomous Vehicles](https://mlsanity.com/paper/2008.09510/) (2020)
- [Operation is the hardest teacher: estimating DNN accuracy looking for mispredictions](https://mlsanity.com/paper/2102.04287/) (2021)
- [Detecting Operational Adversarial Examples for Reliable Deep Learning](https://mlsanity.com/paper/2104.06015/) (2021)
- [Assurance 2.0: A Manifesto](https://mlsanity.com/paper/2004.10474/) (2020)
- [Distribution Awareness for AI System Testing](https://mlsanity.com/paper/2105.02540/) (2021)
- [Quantifying Assurance in Learning-enabled Systems](https://mlsanity.com/paper/2006.10345/) (2020)
- [Safety Case Templates for Autonomous Systems](https://mlsanity.com/paper/2102.02625/) (2021)

## Built on by (most cited first, 16 of 16)

- [Building Guardrails for Large Language Models](https://mlsanity.com/paper/2402.01822/) (2024)
- [SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability](https://mlsanity.com/paper/2208.09418/) (2022)
- [ProTIP: Probabilistic Robustness Verification on Text-to-Image Diffusion Models against Stochastic Perturbation](https://mlsanity.com/paper/2402.15429/) (2024)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey](https://mlsanity.com/paper/2409.18214/) (2024)
- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [Investigating the Corruption Robustness of Image Classifiers with Random Lp-norm Corruptions](https://mlsanity.com/paper/2305.05400/) (2023)
- [Hierarchical Testing with Rabbit Optimization for Industrial Cyber-Physical Systems](https://mlsanity.com/paper/2507.04100/) (2025)
- [Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems](https://mlsanity.com/paper/2210.14991/) (2022)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [SCOPE: Sequential Conformal Probing for Reliable OOD Rejection in LLM Services](https://mlsanity.com/paper/2606.21255/) (2026)
- [PRBench: A Standardized Probabilistic Robustness Benchmark](https://mlsanity.com/paper/2511.01724/) (2025)
- [TRACE: A Metrologically-Grounded Engineering Framework for Trustworthy Agentic AI Systems in Operationally Critical Domains](https://mlsanity.com/paper/2605.03838/) (2026)
- [Towards Continuous Assurance with Formal Verification and Assurance Cases](https://mlsanity.com/paper/2511.14805/) (2025)
- [Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents](https://mlsanity.com/paper/2609.39717/) (2026)

## Abstract (arXiv)

The increasing use of Machine Learning (ML) components embedded in autonomous systems -- so-called Learning-Enabled Systems (LESs) -- has resulted in the pressing need to assure their functional safety. As for traditional functional safety, the emerging consensus within both, industry and academia, is to use assurance cases for this purpose. Typically assurance cases support claims of reliability in support of safety, and can be viewed as a structured way of organising arguments and evidence generated from safety analysis and reliability modelling activities. While such assurance activities are traditionally guided by consensus-based standards developed from vast engineering experience, LESs pose new challenges in safety-critical application due to the characteristics and design of ML models. In this article, we first present an overall assurance framework for LESs with an emphasis on quantitative aspects, e.g., breaking down system-level safety targets to component-level requirements and supporting claims stated in reliability metrics. We then introduce a novel model-agnostic Reliability Assessment Model (RAM) for ML classifiers that utilises the operational profile and robustness verification evidence. We discuss the model assumptions and the inherent challenges of assessing ML reliability uncovered by our RAM and propose solutions to practical use. Probabilistic safety argument templates at the lower ML component-level are also developed based on the RAM. Finally, to evaluate and demonstrate our methods, we not only conduct experiments on synthetic/benchmark datasets but also scope our methods with case studies on simulated Autonomous Underwater Vehicles and physical Unmanned Ground Vehicles.