# Reliability Engineering for AI Systems: Challenges, Methods, and Directions

arXiv: https://arxiv.org/abs/2609.35316 · HTML: https://mlsanity.com/paper/2609.35316/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024)
- [Memory Injection Attacks on LLM Agents via Query-Only Interaction](https://mlsanity.com/paper/2503.03704/) (2025)
- [Towards a Science of AI Agent Reliability](https://mlsanity.com/paper/2602.16666/) (2026)
- [AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocations](https://mlsanity.com/paper/2603.10749/) (2026)
- [STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds](https://mlsanity.com/paper/2506.02009/) (2025)
- [The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement](https://mlsanity.com/paper/2609.11873/) (2026)
- [On the Reliability Limits of LLM-Based Multi-Agent Planning](https://mlsanity.com/paper/2603.26993/) (2026)
- [Bounded Agents: Delegation Security for Multi-Agent AI Systems](https://mlsanity.com/paper/2608.15888/) (2026)
- [Emergent Risks in Generative Multi-Agent Systems](https://mlsanity.com/paper/2603.27771/) (2026)
- [Real-Time Detection and Repair of LLM Agent Failures](https://mlsanity.com/paper/2608.02464/) (2026)
- [AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection](https://mlsanity.com/paper/2608.06790/) (2026)
- [Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations](https://mlsanity.com/paper/2608.11323/) (2026)
- [MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance](https://mlsanity.com/paper/2608.21867/) (2026)
- [MemArchitect: A Policy Driven Memory Governance Layer](https://mlsanity.com/paper/2603.18330/) (2026)
- [Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability](https://mlsanity.com/paper/2605.10516/) (2026)
- [Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence](https://mlsanity.com/paper/2608.12895/) (2026)
- [One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows](https://mlsanity.com/paper/2608.19741/) (2026)
- [Gaussian Derivative Change-point Detection for Early Warnings of Industrial System Failures](https://mlsanity.com/paper/2410.22594/) (2024)
- [The Agent Incident Registry: Toward Preventing Repeated AI Agent Failures](https://mlsanity.com/paper/2609.11030/) (2026)
- [Boost-R: Gradient Boosted Trees for Recurrence Data](https://mlsanity.com/paper/2107.08784/) (2021)
- [OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality](https://mlsanity.com/paper/2608.05263/) (2026)
- [When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry](https://mlsanity.com/paper/2608.14680/) (2026)
- [SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems](https://mlsanity.com/paper/2609.00595/) (2026)

## Abstract (arXiv)

AI reliability concerns whether an AI system performs its intended function dependably over a stated period and under stated operating conditions, with stated evidence. As these systems become more autonomous, that function includes more than a correct output. Retrieval, memory, tool use, permissions, human oversight, and interactions among systems must operate consistently and safely, and, for generative systems, so must the reasoning process that produces the output. Average benchmark accuracy measures capability; it does not quantify this broader reliability claim. This paper adapts established reliability engineering methods, from failure definitions and operational envelopes to FMEA, accelerated testing, field monitoring, and reliability growth, to AI systems. A four-level diagnostic framework classifies failures as component, operational-loop, agentic-conduct, or network and governance failures. Test, evaluation, verification, and validation (TEVV), sequential monitoring, and FRACAS create and refresh evidence. SMART provides statistical guidance for measurement, analysis, assessment, and test planning; the NIST AI Risk Management Framework provides organizational guidance for governance, evaluation, monitoring, and mitigation. Three cases illustrate the program: adversarial testing of a convolutional neural network, perception-error propagation, and autonomous-vehicle disengagements. Established reliability engineering provides a usable foundation; new measurements and safety guardrails are still needed as these systems are self-evolving.