mlsanityData to 14 Oct 2026

From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection

arXiv 2412.10198 ↗First posted 13 Dec 2024cs.CR cs.AIcited by 38

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Tool-calling has changed Large Language Model (LLM) applications by integrating external tools, significantly enhancing their functionality across diverse tasks. However, this integration also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. To fill this gap, we present ToolCommander, a novel framework designed to exploit vulnerabilities in LLM tool-calling systems through adversarial tool injection. Our framework employs a well-designed two-stage attack strategy. Firstly, it injects malicious tools to collect user queries, then dynamically updates the injected tools based on the stolen information to enhance subsequent attacks. These stages enable ToolCommander to execute privacy theft, launch denial-of-service attacks, and even manipulate business competition by triggering unscheduled tool-calling. Notably, the ASR reaches 91.67% for privacy theft and hits 100% for denial-of-service and unscheduled tool calling in certain cases. Our work demonstrates that these vulnerabilities can lead to severe consequences beyond simple misuse of tool-calling systems, underscoring the urgent need for robust defensive strategies to secure LLM Tool-calling systems.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 28 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

22 of 28 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  12. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
10 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  2. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  3. 2023Detecting Language Model Attacks with Perplexitycited by 431
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  5. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  6. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  7. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculumcited by 65
  8. 2024Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learningcited by 51
  9. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28
  10. 2023CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Executioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 23.

38 papers.

Sort:

By area:

  1. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  4. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  5. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 24
  6. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
  7. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 18
  8. 2026AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystemscited by 17
  9. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 17
  10. 2025Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 16
  11. 2025A Vision for Access Control in LLM-based Agent Systemscited by 14
  12. 2026MCP-ITP: An Automated Framework for Implicit Tool Poisoning in MCPcited by 14
26 more
  1. 2026When Lower Privileges Suffice: Investigating Over-Privileged Tool Selection in LLM Agentscited by 13
  2. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 11
  3. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  4. 2025Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning Systemcited by 10
  5. 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 9
  6. 2026MemMorph: Tool Hijacking in LLM Agents via Memory Poisoningcited by 9
  7. 2026AgentRaft: Automated Detection of Data Over-Exposure in LLM Agentscited by 6
  8. 2026Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environmentscited by 6
  9. 2025Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessmentcited by 5
  10. 2025Across Programming Language Silos: A Study on Cross-Lingual Retrieval-augmented Code Generationcited by 3
  11. 2026Agents at Risk: How Users Unwittingly Undermine LLM Safetycited by 3
  12. 2026Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Searchcited by 2
  13. 2026GAMMAF: A Common Framework for Graph-Based Anomaly Monitoring Benchmarking in LLM Multi-Agent Systemscited by 2
  14. 2025Can LLMs Threaten Human Survival? Benchmarking Potential Existential Threats from LLMs via Prefix Completioncited by 1
  15. 2025Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agentscited by 1
  16. 2026ShieldNet: Network-Level Guardrails against Emerging Supply-Chain Injections in Agentic Systemscited by 1
  17. 2026Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoningcited by 1
  18. 2026Security of OpenClaw Agents: Fundamentals, Attacks, and Countermeasurescited by 1
  19. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  20. 2026Blue Teaming Function-Calling Agentscited by 0
  21. 2026Self-Adaptive Multi-Agent LLM-Based Security Pattern Selection for IoT Systemscited by 0
  22. 2026Agent-Facing Information Design in LLM Tool Registriescited by 0
  23. 2026Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agentscited by 0
  24. 2026The Surface You Test Is Not the Surface That Breakscited by 0
  25. 2026PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noisecited by 0
  26. 2026HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
  2. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection
  3. 2025Parasites in the Toolchain: A Large-Scale Analysis of Attacks on the MCP Ecosystem
  4. 2026From Prompt Injection to Web Exploitation: Revisiting Classic Vulnerabilities in LLM-Integrated Applications
  5. 2024MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistants
  6. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltration
  7. 2026MalTool: Malicious Tool Attacks on LLM Agents
  8. 2026Think Twice Before You Act: Protecting LLM Agents Against Tool Description Poisoning via Isolated Planning