mlsanityData to 7 Oct 2026

Imprompter: Tricking LLM Agents into Improper Tool Use

arXiv 2410.14923 ↗First posted 19 Oct 2024cs.CRcited by 66

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large Language Model (LLM) Agents are an emerging computing paradigm that blends generative machine learning with tools such as code interpreters, web browsing, email, and more generally, external resources. These agent-based systems represent an emerging shift in personal computing. We contribute to the security foundations of agent-based systems and surface a new class of automatically computed obfuscated adversarial prompt attacks that violate the confidentiality and integrity of user resources connected to an LLM agent. We show how prompt optimization techniques can find such prompts automatically given the weights of a model. We demonstrate that such attacks transfer to production-level agents. For example, we show an information exfiltration attack on Mistral's LeChat agent that analyzes a user's conversation, picks out personally identifiable information, and formats it into a valid markdown command that results in leaking that data to the attacker's server. This attack shows a nearly 80% success rate in an end-to-end evaluation. We conduct a range of experiments to characterize the efficacy of these attacks and find that they reliably work on emerging agent-based systems like Mistral's LeChat, ChatGLM, and Meta's Llama. These attacks are multimodal, and we show variants in the text-only and image domains.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 28 of its 57 references that are in the map. First posted on arXiv 19 Oct 2024.

Builds on

28 of 57 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  10. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
16 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  2. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  5. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  6. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  7. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  8. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  9. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  10. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  11. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  12. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  13. 2021Lower Perplexity is Not Always Human-Likecited by 52
  14. 2019Supervised and Unsupervised Neural Approaches to Text Readabilitycited by 45
  15. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 43
  16. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 37; 2026: 29.

66 papers.

Sort:

By area:

  1. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 257
  2. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  3. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  6. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  7. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 81
  8. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  9. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  10. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  11. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  12. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
38 more
  1. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  2. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  3. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  4. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  5. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 28
  6. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  7. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  8. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  9. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25
  10. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 23
  11. 2025AgentBound: Securing Execution Boundaries of AI Agentscited by 20
  12. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  13. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  14. 2026From Storage to Steering: Memory Control Flow Attacks on LLM Agentscited by 17
  15. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  16. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  17. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  18. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  19. 2025ceLLMate: Sandboxing Browser AI Agentscited by 14
  20. 2026Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANPcited by 14
  21. 2026How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Studycited by 14
  22. 2025Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Treecited by 13
  23. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  24. 2025From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systemscited by 12
  25. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  26. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  27. 2025Systems Security Foundations for Agentic Computingcited by 9
  28. 2026HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?cited by 9
  29. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
  30. 2026No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agentscited by 6
  31. 2026INFA-Guard: Mitigating Malicious Propagation via Infection-Aware Safeguarding in LLM-Based Multi-Agent Systemscited by 5
  32. 2026Assessing Automated Prompt Injection Attacks in Agentic Environmentscited by 5
  33. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  34. 2025QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systemscited by 4
  35. 2026AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agentscited by 4
  36. 2025FaSTA^*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editingcited by 3
  37. 2025Soft Instruction De-escalation Defensecited by 3
  38. 2026Agent-Fence: Mapping Security Vulnerabilities Across Deep Research Agentscited by 3

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
  2. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach
  3. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
  4. 2024Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation
  5. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  6. 2025Unveiling Privacy Risks in LLM Agent Memory
  7. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  8. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution