mlsanityData to 14 Oct 2026

Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage

arXiv 2412.15606 ↗First posted 20 Dec 2024cs.AI cs.CVcited by 51

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The advancement of large language models (LLMs) prompts the development of multi-modal agents, which are used as a controller to call external tools, providing a feasible way to solve practical tasks. In this paper, we propose a multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning. To preserve the data quality, we prompt the GPT-4o mini model to generate queries, files, and trajectories, followed by query-file and trajectory verifiers. Based on the data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks with trajectories of tool usage. Then, we develop the T3-Agent via underlineTrajectory underlineTuning on VLMs for underlineTool usage using MM-Traj. Evaluations on the GTA and GAIA benchmarks show that the T3-Agent consistently achieves improvements on two popular VLMs: MiniCPM-V-8.5B and Qwen2-VL-7B, which outperforms untrained VLMs by 20%, showing the effectiveness of the proposed data synthesis pipeline, leading to high-quality data for tool-usage capabilities.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 28 of its 71 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

28 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets26 Jun
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun

Other references · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  11. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  12. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
14 more
  1. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 182
  2. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  3. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  4. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  5. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 120
  6. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  7. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  8. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 69
  9. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  10. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  11. 2024Sibyl: Simple yet Effective Agent Framework for Complex Real-world Reasoningcited by 18
  12. 2024Details Make a Difference: Object State-Sensitive Neurorobotic Task Planningcited by 7
  13. 2023VIoTGPT: Learning to Schedule Vision Tools in LLMs towards Intelligent Video Internet of Thingscited by 4
  14. 2024Layout Generation Agents with Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 20.

51 papers.

Sort:

By area:

  1. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 96
  2. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
  3. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
  4. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 26
  5. 2025Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agentscited by 24
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  7. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  8. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 19
  9. 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
  10. 2025ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memorycited by 14
  11. 2025ToolVQA: A Dataset for Multi-step Reasoning VQA with External Toolscited by 13
  12. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoningcited by 13
38 more
  1. 2025Texture or Semantics? Vision-Language Models Get Lost in Font Recognitioncited by 11
  2. 2025ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Modelcited by 11
  3. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 9
  4. 2025DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuningcited by 8
  5. 2024Multi-Modal Data Exploration via Language Agentscited by 7
  6. 2025Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitivescited by 7
  7. 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 7
  8. 2026VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authoritycited by 6
  9. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  10. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scalescited by 5
  11. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  12. 2026Anticipatory Planning for Multimodal AI Agentscited by 4
  13. 2026GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotationcited by 4
  14. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 4
  15. 2025Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Modelscited by 3
  16. 2025Multi-Step Reasoning for Embodied Question Answering via Tool Augmentationcited by 3
  17. 2026TIR-Flow: Active Video Search and Reasoning with Frozen VLMscited by 3
  18. 2026LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathologycited by 3
  19. 2026Retrieval-Augmented Robots via Retrieve-Reason-Actcited by 2
  20. 2025General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Actingcited by 1
  21. 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuningcited by 1
  22. 2025Long-Horizon Visual Imitation Learning via Plan and Code Reflectioncited by 1
  23. 2025Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environmentscited by 1
  24. 2025MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoningcited by 1
  25. 2025City Navigation in the Wild: Exploring Emergent Navigation from Web-Scale Knowledge in MLLMscited by 1
  26. 2026PerfGuard: A Performance-Aware Agent for Visual Content Generationcited by 1
  27. 2026Can LLM Agents Sustain Long-Horizon Organizational Dynamics?cited by 1
  28. 2026Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editingcited by 1
  29. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capabilitycited by 1
  30. 2026Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-usecited by 1
  31. 2025Allen: Rethinking MAS Design through Step-Level Policy Autonomycited by 0
  32. 2025GPU Memory Prediction for Multimodal Model Trainingcited by 0
  33. 2026Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviorscited by 0
  34. 2026Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perceptioncited by 0
  35. 2026PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Frameworkcited by 0
  36. 2026ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agentscited by 0
  37. 2026VTO: Visual Tool Orchestration for Video Anomaly Detectioncited by 0
  38. 2026JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agentscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis
  2. 2024Learning to Use Tools via Cooperative and Interactive Agents
  3. 2026UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents
  4. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
  5. 2025Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects
  6. 2026Internalizing Tool Knowledge in Small Language Models via QLoRA Fine-Tuning
  7. 2025AutoTool: Efficient Tool Selection for Large Language Model Agents
  8. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration