mlsanityData to 7 Oct 2026

The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems

arXiv 2607.19292 ↗First posted 21 Jul 2026cs.CY cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Current AI safety discourse still focuses disproportionately on visible failures, including obvious harms, dramatic misuse, and hypothetical catastrophic scenarios. That focus is incomplete. In deployed systems, many of the most consequential failures are quieter: plausible rather than spectacular, distributed across components rather than localized in a single output, and normalized by workflows before they are recognized as hazards. We argue that a central safety challenge in modern AI systems is increasingly not only whether a model emits a harmful response, but whether the broader socio-technical system preserves the conditions under which errors remain visible, contestable, containable, and recoverable. We propose a five-layer framework for diagnosing these hidden risks: (1) epistemic integrity, concerning whether evidence and uncertainty are represented honestly enough to support calibrated reliance; (2) control integrity, concerning whether authority, permissions, and action boundaries remain robust under attack and optimization; (3) temporal integrity, concerning whether safety holds across sessions, memory updates, and deployment drift; (4) organizational integrity, concerning whether institutions retain the capacity to audit, assign responsibility, and intervene effectively; and (5) ecosystem integrity, concerning whether AI systems preserve rather than erode the information environment on which future oversight depends. Across these layers, we identify under-recognized risk patterns, including overreliance, uncertainty and legitimacy laundering in retrieval, prompt injection, reward hacking, memory poisoning, evaluation deception, fictional human oversight, synthetic evidence pollution, and model collapse. We conclude with design and governance recommendations and a research agenda for shifting AI safety from model-centric evaluation toward socio-technical reliability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 55 of its 86 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

55 of 86 references.

References · 55

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  8. 2023Active Retrieval Augmented Generationcited by 506
  9. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  10. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  11. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  12. 2024Alignment faking in large language modelscited by 316
38 more
  1. 2025Defeating Prompt Injections by Designcited by 256
  2. 2023Self-Consuming Generative Models Go MADcited by 239
  3. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  4. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  5. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  6. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  7. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  8. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  9. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  11. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  12. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  13. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  14. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  15. 2022Explanations Can Reduce Overreliance on AI Systems During Decision-Makingcited by 111
  16. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  17. 2025How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behaviorcited by 95
  18. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  19. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 84
  20. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  21. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  22. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 54
  23. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  24. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  25. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 40
  26. 2023Understanding accountability in algorithmic supply chainscited by 39
  27. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  28. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  29. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 33
  30. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  31. 2021Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systemscited by 30
  32. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  33. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 28
  34. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  35. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  36. 2024A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harmscited by 19
  37. 2024To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Modelscited by 17
  38. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Safety is an Epistemic Property, Not a Behavioral One
  2. 2026The Missing Layer: Specification Infrastructure for AI Oversight ↗
  3. 2026Beyond single-channel agentic benchmarking
  4. 2026AI Deployment Accountability Engineering: A Vision for Accountable AI in Safety-Critical Socio-Technical Systems
  5. 2024AI Risk Management Should Incorporate Both Safety and Security
  6. 2025From Failure Modes to Reliability Awareness in Generative and Agentic AI System
  7. 2025A Safety and Security Framework for Real-World Agentic Systems
  8. 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails