# A Safety Framework for Critical Systems Utilising Deep Neural Networks

arXiv: https://arxiv.org/abs/2003.05311 · HTML: https://mlsanity.com/paper/2003.05311/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2020.

## Built on by (most cited first, 19 of 19)

- [Building Guardrails for Large Language Models](https://mlsanity.com/paper/2402.01822/) (2024)
- [Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance](https://mlsanity.com/paper/2112.00646/) (2021)
- [Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles](https://mlsanity.com/paper/2106.01258/) (2021)
- [Statistical Perspectives on Reliability of Artificial Intelligence Systems](https://mlsanity.com/paper/2111.05391/) (2021)
- [Detecting Operational Adversarial Examples for Reliable Deep Learning](https://mlsanity.com/paper/2104.06015/) (2021)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [Reliability Analysis of Artificial Intelligence Systems Using Recurrent Events Data from Autonomous Vehicles](https://mlsanity.com/paper/2102.01740/) (2021)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2108.10451/) (2021)
- [Safety Case Templates for Autonomous Systems](https://mlsanity.com/paper/2102.02625/) (2021)
- [Unaligned but Safe -- Formally Compensating Performance Limitations for Imprecise 2D Object Detection](https://mlsanity.com/paper/2202.05123/) (2022)
- [Safety Metrics for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2105.10142/) (2021)
- [Logically Sound Arguments for the Effectiveness of ML Safety Measures](https://mlsanity.com/paper/2111.02649/) (2021)
- [Short-term Load Forecasting with Distributed Long Short-Term Memory](https://mlsanity.com/paper/2208.01147/) (2022)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features](https://mlsanity.com/paper/2103.03704/) (2021)
- [DeepCodeProbe: Towards Understanding What Models Trained on Code Learn](https://mlsanity.com/paper/2407.08890/) (2024)
- [Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey](https://mlsanity.com/paper/2111.14324/) (2021)

## Abstract (arXiv)

Increasingly sophisticated mathematical modelling processes from Machine Learning are being used to analyse complex data. However, the performance and explainability of these models within practical critical systems requires a rigorous and continuous verification of their safe utilisation. Working towards addressing this challenge, this paper presents a principled novel safety argument framework for critical systems that utilise deep neural networks. The approach allows various forms of predictions, e.g., future reliability of passing some demands, or confidence on a required reliability level. It is supported by a Bayesian analysis using operational data and the recent verification and validation techniques for deep learning. The prediction is conservative -- it starts with partial prior knowledge obtained from lifecycle activities and then determines the worst-case prediction. Open challenges are also identified.