mlsanityData to 14 Oct 2026

Evil Geniuses: Delving into the Safety of LLM-based Agents

arXiv 2311.11855 ↗First posted 20 Nov 2023cs.CLcited by 106

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Rapid advancements in large language models (LLMs) have revitalized in LLM-based agents, exhibiting impressive human-like behaviors and cooperative capabilities in various scenarios. However, these agents also bring some exclusive risks, stemming from the complexity of interaction environments and the usability of tools. This paper delves into the safety of LLM-based agents from three perspectives: agent quantity, role definition, and attack level. Specifically, we initially propose to employ a template-based attack strategy on LLM-based agents to find the influence of agent quantity. In addition, to address interaction environment and role specificity issues, we introduce Evil Geniuses (EG), an effective attack method that autonomously generates prompts related to the original role to examine the impact across various role definitions and attack levels. EG leverages Red-Blue exercises, significantly improving the generated prompt aggressiveness and similarity to original roles. Our evaluations on CAMEL, Metagpt and ChatDev based on GPT-3.5 and GPT-4, demonstrate high success rates. Extensive evaluation and discussion reveal that these agents are less robust, prone to more harmful behaviors, and capable of generating stealthier content than LLMs, highlighting significant safety challenges and guiding future research. Our code is available at https://github.com/T1aNS1R/Evil-Geniuses.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 28 of its 39 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

28 of 39 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  12. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
16 more
  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  3. 2023Cognitive Architectures for Language Agentscited by 429
  4. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  5. 2023Large Language Models as Tool Makerscited by 303
  6. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  7. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  8. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 192
  9. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 183
  10. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 107
  11. 2023ChatLLM Network: More brains, More intelligencecited by 37
  12. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 35
  13. 2023Towards best practices in AGI safety and governance: A survey of expert opinioncited by 27
  14. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  15. 2023Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industriescited by 19
  16. 2022Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understandingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 39; 2025: 43; 2026: 23.

106 papers.

Sort:

By area:

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  3. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  4. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  6. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  7. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
  8. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
  9. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  10. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 137
  11. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  12. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
38 more
  1. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  2. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  4. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 99
  5. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  6. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
  7. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  8. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  9. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  10. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 64
  11. 2024Large Language Model Safety: A Holistic Surveycited by 59
  12. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 59
  13. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
  14. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  15. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  16. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  17. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 48
  18. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  19. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  20. 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modelingcited by 36
  21. 2025textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attackscited by 34
  22. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 31
  23. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  24. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 29
  25. 2024AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Modelscited by 28
  26. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  27. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  28. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
  29. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 24
  30. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  31. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  32. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Modelscited by 21
  33. 2024Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Modelscited by 20
  34. 2025Rethinking the Reliability of Multi-agent System: A Perspective from Byzantine Fault Tolerancecited by 17
  35. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 16
  36. 2025Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systemscited by 13
  37. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  38. 2024An Optimizable Suffix Is Worth A Thousand Templates: Efficient Black-box Jailbreaking without Affirmative Phrases via LLM as Optimizercited by 11

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
  2. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  3. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  4. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents
  5. 2026Detection of adversarial intent in Human-AI teams using LLMs
  6. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
  7. 2023LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing
  8. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems