mlsanityData to 7 Oct 2026

MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning

arXiv 2401.10727 ↗First posted 19 Jan 2024cs.CVcited by 66

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Recently, the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However, the current LLMs' ability to perceive tool use is limited to a single text query, which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore, in this paper, we propose MLLM-Tool, a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability, we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions, which provides more potential solutions for the same query. The experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at https://github.com/MLLM-Tool/MLLM-Tool.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 48 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

36 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
23 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  13. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  14. 2021AST: Audio Spectrogram Transformercited by 634
  15. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  16. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  17. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  18. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  19. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  20. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  21. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  22. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  23. 2023Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Supportcited by 75

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 30; 2026: 21.

66 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
  3. 2024Large Multimodal Agents: A Surveycited by 99
  4. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  5. 2024GTA: A Benchmark for General Tool Agentscited by 77
  6. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 74
  7. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 49
  8. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  9. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  10. 2024Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generationcited by 35
  11. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  12. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 34
38 more
  1. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  2. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  3. 2025MDCrow: Automating Molecular Dynamics Workflows with Large Language Modelscited by 22
  4. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  5. 2024The Evolution of Multimodal Model Architecturescited by 21
  6. 2025Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agentscited by 20
  7. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  8. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  9. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  10. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIscited by 14
  11. 2025ToolVQA: A Dataset for Multi-step Reasoning VQA with External Toolscited by 13
  12. 2024ChatHuman: Chatting about 3D Humans with Toolscited by 11
  13. 2025MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Sciencecited by 10
  14. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictionscited by 9
  15. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  16. 2025AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memorycited by 8
  17. 2024GUIDE: Graphical User Interface Data for Executioncited by 7
  18. 2026SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glassescited by 6
  19. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  20. 2026Towards Pixel-Level VLM Perception via Simple Points Predictioncited by 5
  21. 2025MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Modelscited by 4
  22. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  23. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understandingcited by 4
  24. 2026Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignmentcited by 4
  25. 2026Anticipatory Planning for Multimodal AI Agentscited by 4
  26. 2025AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agentscited by 3
  27. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
  28. 2024SEAL: Suite for Evaluating API-use of LLMscited by 2
  29. 2025MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understandingcited by 2
  30. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  31. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  32. 2025From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Imagescited by 2
  33. 2025ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learningcited by 2
  34. 2026A Versatile Multimodal Agent for Multimedia Content Generationcited by 2
  35. 2026LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathologycited by 2
  36. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 2
  37. 2025SurgVisAgent: Multimodal Agentic Model for Versatile Surgical Visual Enhancementcited by 1
  38. 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuningcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  2. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  3. 2024Learning to Use Tools via Cooperative and Interactive Agents
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  6. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models
  7. 2025M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
  8. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks