mlsanityData to 7 Oct 2026

ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages

arXiv 2402.10753 ↗First posted 16 Feb 2024cs.CL cs.AIcited by 77

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Tool learning is widely acknowledged as a foundational approach or deploying large language models (LLMs) in real-world scenarios. While current research primarily emphasizes leveraging tools to augment LLMs, it frequently neglects emerging safety considerations tied to their application. To fill this gap, we present *ToolSword*, a comprehensive framework dedicated to meticulously investigating safety issues linked to LLMs in tool learning. Specifically, ToolSword delineates six safety scenarios for LLMs in tool learning, encompassing **malicious queries** and **jailbreak attacks** in the input stage, **noisy misdirection** and **risky cues** in the execution stage, and **harmful feedback** and **error conflicts** in the output stage. Experiments conducted on 11 open-source and closed-source LLMs reveal enduring safety challenges in tool learning, such as handling harmful queries, employing risky tools, and delivering detrimental feedback, which even GPT-4 is susceptible to. Moreover, we conduct further studies with the aim of fostering research on tool learning safety. The data is released in https://github.com/Junjie-Ye/ToolSword.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 31 of its 48 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

31 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning16 Jan
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  5. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  6. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  7. 2023Tool Learning with Foundation Models17 Apr
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
11 more
  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  6. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  7. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  8. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  9. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  10. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  11. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 30; 2026: 35.

77 papers.

Sort:

By area:

  1. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  2. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 257
  3. 2024Tool Learning with Large Language Models: A Surveycited by 221
  4. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  9. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  10. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  11. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  12. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
38 more
  1. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  2. 2024From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injectioncited by 36
  3. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  4. 2026BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoningcited by 25
  5. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  6. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 23
  7. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 21
  8. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Modelscited by 19
  9. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environmentscited by 19
  10. 2025Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systemscited by 17
  11. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  12. 2025ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversationscited by 14
  13. 2024TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Usecited by 11
  14. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  15. 2026Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Usecited by 9
  16. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
  17. 2026SOK: A Taxonomy of Attack Vectors and Defense Strategies for Agentic Supply Chain Runtimecited by 8
  18. 2024When to Speak, When to Abstain: Contrastive Decoding with Abstentioncited by 7
  19. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  20. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
  21. 2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6
  22. 2025Oversight Structures for Agentic AI in Public-Sector Organizationscited by 6
  23. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  24. 2025Formalizing the Safety, Security, and Functional Properties of Agentic AI Systemscited by 5
  25. 2026Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesiscited by 5
  26. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  27. 2024What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworkscited by 4
  28. 2025Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessmentcited by 4
  29. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  30. 2025With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systemscited by 4
  31. 202460 Data Points are Sufficient to Fine-Tune LLMs for Question-Answeringcited by 2
  32. 2025LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Modelscited by 2
  33. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  34. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  35. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2
  36. 2026Agentic Tool Use in Large Language Modelscited by 2
  37. 2026Convergent Detour Hijacking: Task-Preserving Resource Amplification in Skill-Based LLM Agentscited by 2
  38. 2026AgentFlow: A Flow-Centric Policy Language and Framework for Securing LLM Agent Systemscited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  2. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  3. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  4. 2023All Languages Matter: On the Multilingual Safety of Large Language Models
  5. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
  6. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion
  7. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
  8. 2026Understanding and Preserving Safety in Fine-Tuned LLMs