mlsanityData to 7 Oct 2026

GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction

arXiv 2305.18752 ↗First posted 30 May 2023cs.CV cs.CLcited by 245

On the map: LLM agents and tool use → Tool learning and function calling

also in 1 more
Abstract (arXiv)

This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at https://github.com/StevenGrove/GPT4Tools.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 58 of its 72 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

58 of 72 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2023Segment Anythingcited by 8,158
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  14. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  15. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  16. 2022A Generalist Agentcited by 802
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  18. 2023Instruction Tuning with GPT-4cited by 676
  19. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  20. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  23. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  24. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  25. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  26. 2021Internet-Augmented Dialogue Generationcited by 217
  27. 2022A Unified Sequence Interface for Vision Taskscited by 152
  28. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  29. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  30. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  31. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  32. 2020Learning Dynamic Routing for Semantic Segmentationcited by 79
  33. 2022UViM: A Unified Modeling Approach for Vision with Learned Guiding Codescited by 77
  34. 2020End-to-End Object Detection with Fully Convolutional Networkcited by 70
  35. 2021MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understandingcited by 40
  36. 2019TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detectioncited by 36
  37. 2022ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformercited by 36
  38. 2019Learnable Tree Filter for Structure-preserving Feature Transformcited by 25

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 50; 2024: 96; 2025: 69; 2026: 30.

245 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  4. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  7. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  8. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  9. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  10. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  11. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
38 more
  1. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  6. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  7. 2024Tool Learning with Large Language Models: A Surveycited by 221
  8. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  9. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  10. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  11. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  12. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  13. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  14. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  15. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  16. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  17. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  18. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  19. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  20. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  21. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 102
  22. 2024Large Multimodal Agents: A Surveycited by 99
  23. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  24. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  25. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  26. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  27. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  28. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  29. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  30. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  31. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  32. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  33. 2023ChatPose: Chatting about 3D Human Posecited by 71
  34. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  35. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  36. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  37. 2025Skywork R1V: Pioneering Multimodal Reasoning with Chain-of-Thoughtcited by 69
  38. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68

50 most cited of 245 All 245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Visual Instruction Tuning
  2. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  3. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
  4. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
  5. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  6. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  7. 2023On the Performance of Multimodal Language Models
  8. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning