mlsanityData to 7 Oct 2026

Engineering Trustworthy Agentic AI for Critical Systems

arXiv 2607.18548 ↗First posted 20 Jul 2026cs.AI cs.MA cs.SYcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Agentic artificial intelligence systems, capable of autonomous perception, planning, tool use, and multi-step action, are increasingly proposed for critical engineering domains where decisions carry physical, operational, or economic consequences. This survey addresses a gap in current literature by treating trustworthiness, whether agentic behavior can be verified, audited, and trusted under the constraints that engineering practice actually requires, as a first-class engineering property, rather than evaluating agentic AI by task capability alone. The study adopts a trustworthiness model organized around five cross-cutting dimensions: safety and constraint satisfaction; robustness and reliability; transparency and interpretability; accountability and auditability; and privacy and security. This is mapped onto an agentic assurance workflow spanning perception through audit. Building on this foundation, agentic systems architectures, threats, concrete trust mechanisms, and quantitative metrics are surveyed for direct application in agentic systems development and evaluation. These principles are then examined across four constraint-bound engineering domains: power systems, autonomous vehicles/robotics/UAVs, high-performance computing, and communication networks, identifying recurring design patterns, shared failure modes, and domain-specific gaps. Synthesizing across those domains, agentic AI trustworthiness is shown to be a single problem, with a path outlined toward a reusable, cross-domain assurance framework analogous to the graded certification regimes used by mature safety-critical engineering fields.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 63 of its 131 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

63 of 131 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustness9 Oct
  2. 2025A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control25 May
  3. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  4. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents9 Oct
  5. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 57

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2017Constrained Policy Optimizationcited by 928
38 more
  1. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  2. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  3. 2019Control Barrier Functions: Theory and Applicationscited by 332
  4. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  5. 2020Deep Learning Enabled Semantic Communication Systemscited by 282
  6. 2023An Overview of Catastrophic AI Riskscited by 214
  7. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  8. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  9. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 95
  10. 2021Semantic Communication Systems for Speech Transmissioncited by 93
  11. 2024AI and Memory Wallcited by 89
  12. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  13. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  14. 2025SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planningcited by 58
  15. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 57
  16. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  17. 2023Can LLMs Follow Simple Rules?cited by 45
  18. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  19. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  20. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 31
  21. 2025Thought Communication in Multiagent Collaborationcited by 28
  22. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 22
  23. 2024Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Frameworkcited by 16
  24. 2024Towards Intent-Based Network Management: Large Language Models for Intent Extraction in 5G Core Networkscited by 15
  25. 2025GridMind: LLMs-Powered Agents for Power System Analysis and Operationscited by 15
  26. 2025AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defensescited by 14
  27. 2025Byzantine-Robust Decentralized Coordination of LLM Agentscited by 13
  28. 2025X-GridAgent: An LLM-Powered Agentic AI System for Assisting Power Grid Analysiscited by 10
  29. 2026α3-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networkscited by 10
  30. 2020Securing Vehicle-to-Everything (V2X) Communication Platformscited by 9
  31. 2021The Black-Box Simplex Architecture for Runtime Assurance of Autonomous CPScited by 8
  32. 2021Runtime Safety Assurance for Learning-enabled Control of Autonomous Driving Vehiclescited by 8
  33. 2024SAFE-RL: Saliency-Aware Counterfactual Explainer for Deep Reinforcement Learning Policiescited by 8
  34. 2025Agentic AI Home Energy Management System: A Large Language Model Framework for Residential Load Schedulingcited by 7
  35. 2023SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systemscited by 5
  36. 2024ControlLoc: Physical-World Hijacking Attack on Visual Perception in Autonomous Drivingcited by 5
  37. 2024SlowPerception: Physical-World Latency Attack against Visual Perception in Autonomous Drivingcited by 3
  38. 2024A First Physical-World Trajectory Prediction Attack via LiDAR-induced Deceptions in Autonomous Drivingcited by 3

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TRACE: A Metrologically-Grounded Engineering Framework for Trustworthy Agentic AI Systems in Operationally Critical Domains
  2. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
  3. 2026Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI
  4. 2026Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges
  5. 2026Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
  6. 2026Trustworthy AI Software Engineers
  7. 2026Beyond Component Testing: Validating Agentic AI Systems
  8. 2026Beyond single-channel agentic benchmarking