mlsanityData to 14 Oct 2026

A Survey on Failure Analysis and Fault Injection in AI Systems

arXiv 2407.00125 ↗First posted 28 Jun 2024cs.SE cs.AI cs.DCcited by 10

Not on the map. Why?

Abstract (arXiv)

The rapid advancement of Artificial Intelligence (AI) has led to its integration into various areas, especially with Large Language Models (LLMs) significantly enhancing capabilities in Artificial Intelligence Generated Content (AIGC). However, the complexity of AI systems has also exposed their vulnerabilities, necessitating robust methods for failure analysis (FA) and fault injection (FI) to ensure resilience and reliability. Despite the importance of these techniques, there lacks a comprehensive review of FA and FI methodologies in AI systems. This study fills this gap by presenting a detailed survey of existing FA and FI approaches across six layers of AI systems. We systematically analyze 160 papers and repositories to answer three research questions including (1) what are the prevalent failures in AI systems, (2) what types of faults can current FI tools simulate, (3) what gaps exist between the simulated faults and real-world failures. Our findings reveal a taxonomy of AI system failures, assess the capabilities of existing FI tools, and highlight discrepancies between real-world and simulated failures. Moreover, this survey contributes to the field by providing a framework for fault diagnosis, evaluating the state-of-the-art in FI, and identifying areas for improvement in FI techniques to enhance the resilience of AI systems.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 55 of its 220 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

55 of 220 references.

References · 55

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  8. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  11. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
38 more
  1. 2019HopSkipJumpAttack: A Query-Efficient Decision-Based Attackcited by 333
  2. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  3. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
  4. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  5. 2021Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspectivecited by 116
  6. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 112
  7. 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
  8. 2019Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloadscited by 79
  9. 2018Scikit-Multiflow: A Multi-output Streaming Frameworkcited by 70
  10. 2021Software Engineering for AI-Based Systems: A Surveycited by 69
  11. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 67
  12. 2020GeoDA: a geometric framework for black-box adversarial attackscited by 66
  13. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
  14. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  15. 2023Data Quality for Software Vulnerability Datasetscited by 51
  16. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  17. 2021Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenterscited by 31
  18. 2022Toward Understanding Deep Learning Framework Bugscited by 26
  19. 2021Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlowcited by 25
  20. 2023A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRUcited by 25
  21. 2020SNIFF: Reverse Engineering of Neural Networks with Fault Attackscited by 23
  22. 2020TensorFI: A Flexible Fault Injection Framework for TensorFlow Applicationscited by 22
  23. 2019Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classificationcited by 19
  24. 2023PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Modelscited by 19
  25. 2021Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approachcited by 18
  26. 2023I See Dead People: Gray-Box Adversarial Attack on Image-To-Text Modelscited by 17
  27. 2022On the Impact of Temporal Concept Drift on Model Explanationscited by 14
  28. 2021Understanding Performance Problems in Deep Learning Systemscited by 12
  29. 2018On the Resilience of RTL NN Accelerators: Fault Characterization and Mitigationcited by 9
  30. 2022Demystifying Dependency Bugs in Deep Learning Stackcited by 9
  31. 2022Towards Understanding the Faults of JavaScript-Based Deep Learning Systemscited by 6
  32. 2018SoK: Sanitizing for Securitycited by 5
  33. 2020On the Performance of Hybrid Search Strategies for Systematic Literature Reviews in Software Engineeringcited by 4
  34. 2020Failures and Fixes: A Study of Software System Incident Responsecited by 4
  35. 2021Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspectivecited by 4
  36. 2022enpheeph: A Fault Injection Framework for Spiking and Compressed Deep Neural Networkscited by 4
  37. 2023Mutation Testing of Deep Reinforcement Learning Based on Real Faultscited by 4
  38. 2020Fault Injectors for TensorFlow: Evaluation of the Impact of Random Hardware Faults on Deep CNNscited by 3

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injectioncited by 5
  2. 2026AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systemscited by 3
  3. 2026Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMscited by 3
  4. 2026LIDL: LLM Integration Defect Localization via Knowledge Graph-Enhanced Multi-Agent Analysiscited by 2
  5. 2026SkillForge: Self-Distilling Agents for Project-Specific Issue Resolutioncited by 2
  6. 2026UModel: An Agent-Ready Observability Data Modeling Method at Scalecited by 1
  7. 2026Automated Hardware Validation Test Plan Generation for Large Scale AI Datacenter Platforms Using a Generative AI Multi-Agents Architecturecited by 0
  8. 2026Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidentscited by 0
  9. 2026When Log Parsing Removes What Matters: Detecting Failures in Distributed GPU Workloadscited by 0
  10. 2026Actionable Meta-Requirements for AI Robustness Auditingcited by 0