mlsanityData to 7 Oct 2026

Tool Unlearning for Tool-Augmented LLMs

arXiv 2502.01083 ↗First posted 3 Feb 2025cs.LG cs.AI cs.CLcited by 17

On the map: LLM agents and tool use → Tool learning and function calling

also in 1 more
Abstract (arXiv)

Tool-augmented large language models (LLMs) are often trained on datasets of query-response pairs, which embed the ability to use tools or APIs directly into the parametric knowledge of LLMs. Tool-augmented LLMs need the ability to forget learned tools due to security vulnerabilities, privacy regulations, or tool deprecations. However, ``tool unlearning'' has not been investigated in unlearning literature. We introduce this novel task, which requires addressing distinct challenges compared to traditional unlearning: knowledge removal rather than forgetting individual samples, the high cost of optimizing LLMs, and the need for principled evaluation metrics. To bridge these gaps, we propose ToolDelete, the first approach for unlearning tools from tool-augmented LLMs. It implements three key properties to address the above challenges for effective tool unlearning and introduces a new membership inference attack (MIA) model for effective evaluation. Extensive experiments on multiple tool learning datasets and tool-augmented LLMs show that ToolDelete effectively unlearns randomly selected tools, while preserving the LLM's knowledge on non-deleted tools and maintaining performance on general tasks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 57 of its 70 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

57 of 70 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs23 Feb
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  4. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb
  7. 2022TALM: Tool Augmented Language Models24 May

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2022Editing Models with Task Arithmeticcited by 1,076
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
38 more
  1. 2021Membership Inference Attacks From First Principlescited by 839
  2. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  3. 2023Erasing Concepts from Diffusion Modelscited by 530
  4. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  6. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  7. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  8. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  9. 2020Amnesiac Machine Learningcited by 305
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  12. 2023Large Language Model Unlearningcited by 288
  13. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  14. 2023Towards Unbounded Machine Unlearningcited by 261
  15. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  16. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  17. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  18. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  19. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  20. 2021Machine Unlearning via Algorithmic Stabilitycited by 128
  21. 2022Zero-Shot Machine Unlearningcited by 125
  22. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  23. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  24. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  25. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 94
  26. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  27. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 81
  28. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  29. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  30. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  31. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  32. 2023Shared Adversarial Unlearning: Backdoor Mitigation by Unlearning Shared Adversarial Examplescited by 40
  33. 2023Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projectioncited by 37
  34. 2024Towards Certified Unlearning for Deep Neural Networkscited by 28
  35. 2023MultiDelete for Multimodal Machine Unlearningcited by 20
  36. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearningcited by 16
  37. 2022Adversarial Unlearning: Reducing Confidence Along Adversarial Directionscited by 14
  38. 2023Exploring the Impact of Model Scaling on Parameter-Efficient Tuningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 8.

17 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  3. 2025Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigationcited by 11
  4. 2025Speech Unlearningcited by 7
  5. 2026CURE:Circuit-Aware Unlearning for LLM-based Recommendationcited by 6
  6. 2026Investigating Tool-Memory Conflicts in Tool-Augmented LLMscited by 5
  7. 2025FROG: Fair Removal on Graphscited by 4
  8. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  9. 2025FUTURE: Flexible Unlearning for Tree Ensemblecited by 3
  10. 2025A Survey on Unlearning in Large Language Modelscited by 2
  11. 2026A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearningcited by 2
  12. 2026TRACER: Token ReAssignment for Concept ERasure in Generative Recommendationcited by 2
5 more
  1. 2025Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 0
  2. 2026EvoMU: Evolutionary Machine Unlearningcited by 0
  3. 2026Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 0
  4. 2026Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agentscited by 0
  5. 2026Trajectory Unlearning on LLM-based Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Parameter-Efficient Unlearning for LLMs
  2. 2026The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning
  3. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification
  4. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages
  5. 2024Rethinking Machine Unlearning for Large Language Models
  6. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs
  7. 2024UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs
  8. 2024LLM With Tools: A Survey