mlsanityData to 14 Oct 2026

From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI Agents

arXiv 2603.07496 ↗First posted 8 Mar 2026cs.CR cs.AIcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Artificial Intelligence (AI) agents have evolved from passive predictive tools into active entities capable of autonomous decision-making and environmental interaction, driven by the reasoning capabilities of Large Language Models (LLMs). However, this evolution has introduced critical security vulnerabilities that existing frameworks fail to address. The Hierarchical Autonomy Evolution (HAE) framework organizes agent security into three tiers: Cognitive Autonomy (L1) targets internal reasoning integrity; Execution Autonomy (L2) covers tool-mediated environmental interaction; Collective Autonomy (L3) addresses systemic risks in multi-agent ecosystems. We present a taxonomy of threats spanning cognitive manipulation, physical environment disruption, and multi-agent systemic failures, and evaluate existing defenses while identifying key research gaps. The findings aim to guide the development of multilayered, autonomy-aware defense architectures for trustworthy AI agent systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 67 of its 87 references that are in the map. First posted on arXiv 8 Mar 2026.

Builds on

67 of 87 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026Towards Verifiably Safe Tool Use for LLM Agents12 Jan
  2. 2025When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms9 Nov
  3. 2025AgentGuard: Runtime Verification of AI Agents28 Sep
  4. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents21 Aug
  5. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures24 Jun
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr
  7. 2025Memory Injection Attacks on LLM Agents via Query-Only Interaction5 Mar
  8. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents27 Feb
  9. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attacks20 Feb
  10. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  11. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  12. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
7 more
  1. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  2. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  4. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganography12 Feb
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  7. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 48

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
36 more
  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  3. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  6. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  9. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  10. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  11. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  12. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  13. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  14. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  15. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  16. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  17. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 178
  18. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  19. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  20. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  21. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  22. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 124
  23. 2024Emergent social conventions and collective bias in LLM populationscited by 123
  24. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 99
  25. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 93
  26. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  27. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  28. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  29. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applicationscited by 59
  30. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 50
  31. 2024FlowMind: Automatic Workflow Generation with LLMscited by 30
  32. 2024Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agentscited by 26
  33. 2024ConfusedPilot: Confused Deputy Risks in RAG-based LLMscited by 22
  34. 2026The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societiescited by 7
  35. 2024Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attackscited by 5
  36. 2025Energy-Latency Attacks: A New Adversarial Threat to Deep Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
  2. 2025Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations
  3. 2026Separating Capability from Permission: A Governance Framework for Agentic AI Autonomy Levels
  4. 2025Toward a Safe Internet of Agents
  5. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
  6. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
  7. 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents
  8. 2026AgentWard: A Lifecycle Security Architecture for Autonomous AI Agents