mlsanityData to 7 Oct 2026

Large Multimodal Agents: A Survey

arXiv 2402.15116 ↗First posted 23 Feb 2024cs.CV cs.AI cs.CLcited by 99

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have achieved superior performance in powering text-based AI agents, endowing them with decision-making and reasoning abilities akin to humans. Concurrently, there is an emerging research trend focused on extending these LLM-powered AI agents into the multimodal domain. This extension enables AI agents to interpret and respond to diverse multimodal user queries, thereby handling more intricate and nuanced tasks. In this paper, we conduct a systematic review of LLM-driven multimodal agents, which we refer to as large multimodal agents ( LMAs for short). First, we introduce the essential components involved in developing LMAs and categorize the current body of research into four distinct types. Subsequently, we review the collaborative frameworks integrating multiple LMAs , enhancing collective efficacy. One of the critical challenges in this field is the diverse evaluation methods used across existing studies, hindering effective comparison among different LMAs . Therefore, we compile these evaluation methodologies and establish a comprehensive framework to bridge the gaps. This framework aims to standardize evaluations, facilitating more meaningful comparisons. Concluding our review, we highlight the extensive applications of LMAs and propose possible future research directions. Our discussion aims to provide valuable insights and guidelines for future research in this rapidly evolving field. An up-to-date resource list is available at https://github.com/jun0wanan/awesome-large-multimodal-agents.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 72 of its 123 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

72 of 123 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  2. 2023Large Language Models are Visual Reasoning Coordinators23 Oct
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  4. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  6. 2022Visual Programming: Compositional visual reasoning without training18 Nov

Other references · 66

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Segment Anythingcited by 8,158
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
38 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2023Cognitive Architectures for Language Agentscited by 422
  3. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  4. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  5. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  6. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  7. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  8. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  9. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  10. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  12. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  13. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  14. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  15. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  16. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  17. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  18. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  19. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  20. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  21. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  22. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  23. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  24. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  25. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  26. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  27. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  28. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  29. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  30. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  31. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  32. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  33. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  34. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  35. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  36. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  37. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  38. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 56

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 48; 2026: 32.

99 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2025Memory in the Age of AI Agentscited by 285
  3. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  4. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  8. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  9. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  10. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  11. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  12. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
38 more
  1. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  2. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  3. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 42
  4. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  6. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  7. 2025Harnessing Vision Models for Time Series Analysis: A Surveycited by 26
  8. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  9. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  10. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  11. 2024Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Surveycited by 22
  12. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  13. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  14. 2024BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysiscited by 20
  15. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  16. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  17. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  18. 2024Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehensioncited by 17
  19. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 17
  20. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 16
  21. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 14
  22. 2025Perception in Reflectioncited by 13
  23. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  24. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  25. 2025DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Drivingcited by 11
  26. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  27. 2025MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Sciencecited by 10
  28. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  29. 2026Atlas: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoningcited by 10
  30. 2026MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memorycited by 10
  31. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
  32. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  33. 2025LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documentscited by 7
  34. 2024Rapid and Automated Alloy Design with Graph Neural Network-Powered LLM-Driven Multi-Agent Systemscited by 6
  35. 2025Unlocking the Potential of Generative AI through Neuro-Symbolic Architectures: Benefits and Limitationscited by 6
  36. 2025FieldWorkArena: Agentic AI Benchmark for Real Field Work Taskscited by 6
  37. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  38. 2026SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glassescited by 6

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  2. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
  3. 2025Literature Review Of Multi-Agent Debate For Problem-Solving
  4. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Survey
  6. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Review
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges
  8. 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning