mlsanityData to 14 Oct 2026

LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

arXiv 2311.05437 ↗First posted 9 Nov 2023cs.CV cs.AI cs.CLcited by 193

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models. It maintains a skill repository of pre-trained vision and vision-language models and can activate relevant tools based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on multimodal instruction-following data to acquire the ability to use tools, covering visual understanding, generation, external knowledge retrieval, and compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in existing capabilities and exhibits new ones. It is distinct in that the image query is directly grounded and actively engaged throughout the entire human-AI interaction sessions, significantly improving tool use performance and enabling new scenarios.

Key facts. Cited by 193 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 52 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

46 of 52 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  2. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  5. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  6. 2023JourneyDB: A Benchmark for Generative Image Understanding3 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023Generating Images with Multimodal Language Models26 May
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
24 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  4. 2023Instruction Tuning with GPT-4cited by 686
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  7. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  8. 2023Segment Everything Everywhere All at Oncecited by 548
  9. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  11. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  12. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  13. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  14. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  15. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  16. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  17. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  18. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  19. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  20. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  21. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  22. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  23. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 97
  24. 2023ICDAR 2023 Competition on Hierarchical Text Detection and Recognitioncited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 67; 2025: 79; 2026: 41.

193 papers.

Sort:

By area:

  1. 2024Large Language Models: A Surveycited by 525
  2. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  3. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  4. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  5. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  7. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
  8. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  9. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  10. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  11. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  12. 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learningcited by 163
38 more
  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  2. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  3. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  4. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  5. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  6. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  7. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  8. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  9. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  10. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  11. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 109
  12. 2024Large Multimodal Agents: A Surveycited by 105
  13. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  14. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  15. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  16. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 90
  17. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
  18. 2024GTA: A Benchmark for General Tool Agentscited by 77
  19. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  20. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 70
  21. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  22. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  23. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 68
  24. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensingcited by 57
  25. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  26. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 51
  27. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  28. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  29. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 49
  30. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 43
  31. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  32. 2024IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologuescited by 40
  33. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  34. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 38
  35. 2025Reinforced Visual Perception with Toolscited by 38
  36. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 37
  37. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 34
  38. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34

50 most cited of 193 All 193 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-OneVision: Easy Visual Task Transfer
  2. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
  3. 2023Improved Baselines with Visual Instruction Tuning
  4. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  5. 2025LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning
  6. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models
  7. 2025Rethinking Visual Information Processing in Multimodal LLMs
  8. 2025DeepEyesV2: Toward Agentic Multimodal Model