A Survey on Agentic Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
With the recent emergence of revolutionary autonomous agentic systems, research community is witnessing a significant shift from traditional static, passive, and domain-specific AI agents toward more dynamic, proactive, and generalizable agentic AI. Motivated by the growing interest in agentic AI and its potential trajectory toward AGI, we present a comprehensive survey on Agentic Multimodal Large Language Models (Agentic MLLMs). In this survey, we explore the emerging paradigm of agentic MLLMs, delineating their conceptual foundations and distinguishing characteristics from conventional MLLM-based agents. We establish a conceptual framework that organizes agentic MLLMs along three fundamental dimensions: (i) Agentic internal intelligence functions as the system's commander, enabling accurate long-horizon planning through reasoning, reflection, and memory; (ii) Agentic external tool invocation, whereby models proactively use various external tools to extend their problem-solving capabilities beyond their intrinsic knowledge; and (iii) Agentic environment interaction further situates models within virtual or physical environments, allowing them to take actions, adapt strategies, and sustain goal-directed behavior in dynamic real-world scenarios. To further accelerate research in this area for the community, we compile open-source training frameworks, training and evaluation datasets for developing agentic MLLMs. Finally, we review the downstream applications of agentic MLLMs and outline future research directions for this rapidly evolving field. To continuously track developments in this rapidly evolving field, we will also actively update a public repository at https://github.com/HJYao00/Awesome-Agentic-MLLMs.
Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 343 of its 396 references that are in the map. First posted on arXiv 13 Oct 2025.
Builds on
343 of 396 references.
Closest ancestors · 38
Same direction, newest first
- 2025Towards Efficient Multimodal Unified Reasoning Model via Model Merging10 Oct
- 2025Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models15 Sep
- 2025Igniting VLMs toward the Embodied Space15 Sep
- 2025OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoning11 Sep
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning1 Jul
- 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning19 Jun
- 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing11 Jun
- 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior9 Jun
- 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation5 Jun
- 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learning4 Jun
- 2025MiMo-VL Technical Report4 Jun
- 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning2 Jun
26 more
- 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking1 Jun
- 2025ZeroGUI: Automating Online GUI Learning at Zero Human Cost29 May
- 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward22 May
- 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO22 May
- 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs21 May
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
- 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning13 May
- 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning7 May
- 2025Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning25 Apr
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation17 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models10 Apr
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning23 Mar
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
- 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
- 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning10 Mar
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
- 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations20 Nov
- 2024Improve Vision Language Model Chain-of-thought Reasoning21 Oct
- 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding16 Jun
- 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
- 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
Other references · 305
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Visual Instruction Tuningcited by 7,869
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
38 more
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2025Gemma 3 Technical Reportcited by 1,824
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
- 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
- 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
50 most cited of 305 All 305 (JSON)
Built on by
Citing papers per year: 2025: 3; 2026: 20.
23 papers.
Sort:
By area:
- 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 26
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
- 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
- 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 15
- 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systemscited by 9
- 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
- 2026VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoningcited by 8
- 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 6
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
- 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agentscited by 5
- 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaborationcited by 4
11 more
- 2026EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolutioncited by 2
- 2026M3-ACE: Rectifying Visual Perception in Multimodal Math Reasoning via Multi-Agentic Context Engineeringcited by 1
- 2026HippoCamp: Benchmarking Contextual Agents on Personal Computerscited by 1
- 2026Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Disseminationcited by 1
- 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
- 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?cited by 0
- 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Supportcited by 0
- 2026Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generationcited by 0
- 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
- 2026MLLMs Fail to Refuse when Using Tools Agenticallycited by 0
- 2026PLAYGROUND: Progressive Layout Generation with Render-Grounded Planningcited by 0
- 2026PLAYGROUND: Progressive Layout Generation with Render-Grounded Planning13 Oct 2026
- 2026MLLMs Fail to Refuse when Using Tools Agentically2 Oct 2026
- 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs27 Sep 2026
- 2026Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation17 Sep 2026
- 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City27 Aug 2026
- 2026Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination14 Aug 2026
- 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents30 Jun 2026
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI12 Jun 2026
- 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support31 May 2026
- 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaboration1 May 2026
- 2026HippoCamp: Benchmarking Contextual Agents on Personal Computers1 Apr 2026
- 2026M3-ACE: Rectifying Visual Perception in Multimodal Math Reasoning via Multi-Agentic Context Engineering9 Mar 2026
11 more
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline1 Mar 2026
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices25 Feb 2026
- 2026EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolution20 Feb 2026
- 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems17 Feb 2026
- 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories11 Feb 2026
- 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?3 Feb 2026
- 2026VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning29 Jan 2026
- 2026Toward Efficient Agents: Memory, Tool learning, and Planning20 Jan 2026
- 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learning6 Dec 2025
- 2025CaptionQA: Is Your Caption as Useful as the Image Itself?26 Nov 2025
- 2025MedSAM3: Delving into Segment Anything with Medical Concepts24 Nov 2025
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
- 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
- 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Historiescited by 6
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 6
- 2026EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolutioncited by 2
- 2026HippoCamp: Benchmarking Contextual Agents on Personal Computerscited by 1
- 2026HippoCamp: Benchmarking Contextual Agents on Personal Computers1 Apr 2026
- 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline1 Mar 2026
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices25 Feb 2026
- 2026EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolution20 Feb 2026
- 2026DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories11 Feb 2026
- 2026Toward Efficient Agents: Memory, Tool learning, and Planning20 Jan 2026
- 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systemscited by 9
- 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
- 2026VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoningcited by 8
- 2026M3-ACE: Rectifying Visual Perception in Multimodal Math Reasoning via Multi-Agentic Context Engineeringcited by 1
- 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Supportcited by 0
- 2026MLLMs Fail to Refuse when Using Tools Agenticallycited by 0
- 2026MLLMs Fail to Refuse when Using Tools Agentically2 Oct 2026
- 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support31 May 2026
- 2026M3-ACE: Rectifying Visual Perception in Multimodal Math Reasoning via Multi-Agentic Context Engineering9 Mar 2026
- 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems17 Feb 2026
- 2026VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning29 Jan 2026
- 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learning6 Dec 2025
- 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agentscited by 5
- 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaborationcited by 4
- 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
- 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?cited by 0
- 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City27 Aug 2026
- 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents30 Jun 2026
- 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaboration1 May 2026
- 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?3 Feb 2026
- 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 26
- 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 15
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
- 2026Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Disseminationcited by 1
- 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
- 2026PLAYGROUND: Progressive Layout Generation with Render-Grounded Planningcited by 0
- 2026PLAYGROUND: Progressive Layout Generation with Render-Grounded Planning13 Oct 2026
- 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs27 Sep 2026
- 2026Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination14 Aug 2026
- 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI12 Jun 2026
- 2025CaptionQA: Is Your Caption as Useful as the Image Itself?26 Nov 2025
- 2025MedSAM3: Delving into Segment Anything with Medical Concepts24 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2026The Path Ahead for Agentic AI: Challenges and Opportunities
- 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents
- 2026Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
- 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents
- 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
- 2023The Rise and Potential of Large Language Model Based Agents: A Survey
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges