mlsanityData to 7 Oct 2026

VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents

arXiv 2408.06327 ↗First posted 12 Aug 2024cs.AI cs.CL cs.CVcited by 101

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable Visual Foundation Agents. These agents are postulated to excel across a myriad of tasks, potentially approaching general artificial intelligence. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across nine proprietary LMM APIs and eight open models, we demonstrate the considerable yet still developing agent capabilities of these models. Additionally, VAB constructs a trajectory training set constructed through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, promoting substantial performance improvements in LMMs through behavior cloning. Our work not only aims to benchmark existing models but also provides a solid foundation for future development into visual foundation agents. Code, train & test data, and part of fine-tuned open LMMs are available at urlhttps://github.com/THUDM/VisualAgentBench.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 129 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

59 of 129 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  2. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedback12 Oct
  3. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
38 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  14. 2016A Diagram Is Worth A Dozen Imagescited by 793
  15. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  16. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  17. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  18. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  19. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  20. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  21. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  22. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  23. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  24. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  25. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  26. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  27. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  28. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  29. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  30. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  31. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  32. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  33. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  34. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  35. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  36. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  37. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  38. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 52; 2026: 34.

101 papers.

Sort:

By area:

  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  2. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  3. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  5. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  6. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  8. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  9. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  10. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  11. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
38 more
  1. 2025Agent Learning via Early Experiencecited by 79
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  3. 2024Beyond Browsing: API-Based Web Agentscited by 77
  4. 2025LifelongAgentBench: Evaluating LLM Agents as Lifelong Learnerscited by 65
  5. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  6. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  7. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  8. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 49
  9. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  10. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  11. 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
  12. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  13. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  14. 2024Benchmarking Agentic Workflow Generationcited by 36
  15. 2024Diving into Self-Evolving Training for Multimodal Reasoningcited by 31
  16. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  17. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
  18. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  19. 2025Visual Test-time Scaling for GUI Agent Groundingcited by 27
  20. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  21. 2024Advancing Agentic Systems: Dynamic Task Decomposition, Tool Integration and Evaluation using Novel Metrics and Datasetcited by 24
  22. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  23. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21
  24. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  25. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 17
  26. 2025Build the web for agents, not agents for the webcited by 17
  27. 2025Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agentscited by 16
  28. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  29. 2025Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesiscited by 10
  30. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  31. 2025WebOperator: Action-Aware Tree Search for Autonomous Agents in Web Environmentcited by 10
  32. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  33. 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigationcited by 9
  34. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 9
  35. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environmentscited by 8
  36. 2025EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environmentscited by 8
  37. 2025Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Modelscited by 8
  38. 2025FlySearch: Exploring how vision-language models explorecited by 7

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
  2. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
  3. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
  4. 2026EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
  5. 2025DeepEyesV2: Toward Agentic Multimodal Model
  6. 2026MMSkills: Towards Multimodal Skills for General Visual Agents
  7. 2025An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Models
  8. 2024Agent AI: Surveying the Horizons of Multimodal Interaction