# Where AI Assurance Might Go Wrong: Initial lessons from engineering of critical systems

arXiv: https://arxiv.org/abs/2502.03467 · HTML: https://mlsanity.com/paper/2502.03467/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 44 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 5 of 5)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [International Scientific Report on the Safety of Advanced AI (Interim Report)](https://mlsanity.com/paper/2412.05282/) (2024)
- [Automating Semantic Analysis of System Assurance Cases using Goal-directed ASP](https://mlsanity.com/paper/2408.11699/) (2024)
- [Confidence in Assurance 2.0 Cases](https://mlsanity.com/paper/2409.10665/) (2024)

## Built on by (most cited first, 4 of 4)

- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [Threshold Crossings as Tail Events for Catastrophic AI Risk](https://mlsanity.com/paper/2503.18979/) (2025)
- [Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI](https://mlsanity.com/paper/2607.14353/) (2026)
- [Dependable AI-Assisted Engineering: A Formal Framework for AI Participation and Assurance in Safety-Critical Workflows](https://mlsanity.com/paper/2610.04084/) (2026)

## Abstract (arXiv)

We draw on our experience working on system and software assurance and evaluation for systems important to society to summarise how safety engineering is performed in traditional critical systems, such as aircraft flight control. We analyse how this critical systems perspective might support the development and implementation of AI Safety Frameworks. We present the analysis in terms of: system engineering, safety and risk analysis, and decision analysis and support. We consider four key questions: What is the system? How good does it have to be? What is the impact of criticality on system development? and How much should we trust it? We identify topics worthy of further discussion. In particular, we are concerned that system boundaries are not broad enough, that the tolerability and nature of the risks are not sufficiently elaborated, and that the assurance methods lack theories that would allow behaviours to be adequately assured. We advocate the use of assurance cases based on Assurance 2.0 to support decision making in which the criticality of the decision as well as the criticality of the system are evaluated. We point out the orders of magnitude difference in confidence needed in critical rather than everyday systems and how everyday techniques do not scale in rigour. Finally we map our findings in detail to two of the questions posed by the FAISC organisers and we note that the engineering of critical systems has evolved through open and diverse discussion. We hope that topics identified here will support the post-FAISC dialogues.