mlsanityData to 7 Oct 2026

Prompt Injection Attack to Tool Selection in LLM Agents

arXiv 2504.19793 ↗First posted 28 Apr 2025cs.CRcited by 145

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Tool selection is a key component of LLM agents. A popular approach follows a two-step process - retrieval and selection - to pick the most appropriate tool from a tool library for a given task. In this work, we introduce textitToolHijacker, a novel prompt injection attack targeting tool selection in no-box scenarios. ToolHijacker injects a malicious tool document into the tool library to manipulate the LLM agent's tool selection process, compelling it to consistently choose the attacker's malicious tool for an attacker-chosen target task. Specifically, we formulate the crafting of such tool documents as an optimization problem and propose a two-phase optimization strategy to solve it. Our extensive experimental evaluation shows that ToolHijacker is highly effective, significantly outperforming existing manual-based and automated prompt injection attacks when applied to tool selection. Moreover, we explore various defenses, including prevention-based defenses (StruQ and SecAlign) and detection-based defenses (known-answer detection, DataSentinel, perplexity detection, and perplexity windowed detection). Our experimental results indicate that these defenses are insufficient, highlighting the urgent need for developing new defense strategies.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 50 of its 74 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

50 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Defeating Prompt Injections by Design24 Mar
  2. 2024From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection13 Dec
  3. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  4. 2024Dissecting Adversarial Robustness of Multimodal LM Agents18 Jun
  5. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
33 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  6. 2023Detecting Language Model Attacks with Perplexitycited by 411
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  10. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  11. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  12. 2023Large Language Models as Tool Makerscited by 293
  13. 2024Tool Learning with Large Language Models: A Surveycited by 221
  14. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  15. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  16. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  17. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  18. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  19. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  20. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  21. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  22. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  23. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  24. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  25. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  26. 2023Making Language Models Better Tool Learners with Execution Feedbackcited by 69
  27. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  28. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculumcited by 64
  29. 2024ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrievalcited by 40
  30. 2024TOOLVERIFIER: Generalization to New Tools via Self-Verificationcited by 23
  31. 2024Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignmentcited by 18
  32. 2023ProTIP: Progressive Tool Retrieval Improves Planningcited by 12
  33. 2024GeckOpt: LLM System Efficiency via Intent-Based Tool Selectioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 28; 2026: 117.

145 papers.

Sort:

By area:

  1. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  2. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  3. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  4. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
  5. 2025MPMA: Preference Manipulation Attack Against Model Context Protocolcited by 31
  6. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  7. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 31
  8. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  9. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  10. 2026BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoningcited by 25
  11. 2026SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinementcited by 24
  12. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
38 more
  1. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 22
  2. 2026Hidden in Memory: Sleeper Memory Poisoning in LLM Agentscited by 22
  3. 2026Cordon: Semantic Transactions for Tool-Using LLM Agentscited by 19
  4. 2026ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watcherscited by 18
  5. 2026SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skillscited by 18
  6. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
  7. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 17
  8. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  9. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  10. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 15
  11. 2026How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Studycited by 14
  12. 2026When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystemscited by 14
  13. 2025Tool Preferences in Agentic LLMs are Unreliablecited by 13
  14. 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCPcited by 13
  15. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  16. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  17. 2026Detecting Safety Violations Across Many Agent Tracescited by 11
  18. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  19. 2026AIRGuard: Guarding Agent Actions with Runtime Authority Controlcited by 9
  20. 2026Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystemscited by 9
  21. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 8
  22. 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 8
  23. 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 8
  24. 2026MemMorph: Tool Hijacking in LLM Agents via Memory Poisoningcited by 8
  25. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  26. 2025Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perceptioncited by 7
  27. 2026FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systemscited by 7
  28. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  29. 2025Toward a Safe Internet of Agentscited by 6
  30. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 6
  31. 2026Behavioral Integrity Verification for AI Agent Skillscited by 6
  32. 2026Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesiscited by 5
  33. 2026No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skillscited by 5
  34. 2026Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Usecited by 5
  35. 2026Assessing Automated Prompt Injection Attacks in Agentic Environmentscited by 5
  36. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  37. 2026SkillMutator: Benchmarking and Defending Language-and-Code Cross-modal Attacks on LLM Agent Skillscited by 5
  38. 2026Dynamic Malicious Skills in Agentic AIcited by 5

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools
  2. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
  3. 2026Defense Against Indirect Prompt Injection via Tool Result Parsing
  4. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents
  5. 2026Think Twice Before You Act: Protecting LLM Agents Against Tool Description Poisoning via Isolated Planning
  6. 2026AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents
  7. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltration
  8. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection