mlsanityData to 14 Oct 2026

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

arXiv 2401.13919 ↗First posted 25 Jan 2024cs.CL cs.AIcited by 438

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement with human judgment, indicating its effectiveness in providing reliable and accurate assessments of web agents.

Key facts. Cited by 438 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 47 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

32 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
19 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  11. 2019Object Detection in 20 Years: A Surveycited by 531
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  13. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  14. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  15. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  16. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  17. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  18. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  19. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 69; 2025: 161; 2026: 208.

438 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  5. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  6. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  7. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  8. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  9. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  10. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  11. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  12. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
38 more
  1. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
  2. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
  3. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
  4. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  5. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  6. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  7. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  8. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  9. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  10. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  11. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  12. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  13. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  14. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  15. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  16. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 131
  17. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  18. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  19. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  20. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 114
  21. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  22. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
  23. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  24. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  25. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 95
  26. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  27. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 90
  28. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  29. 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
  30. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
  31. 2024Beyond Browsing: API-Based Web Agentscited by 78
  32. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  33. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  34. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
  35. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  36. 2025Interactive Debugging and Steering of Multi-Agent AI Systemscited by 68
  37. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  38. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67

50 most cited of 438 All 438 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
  2. 2026Improving LLM-based Autonomous Web Agents with Filtering
  3. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
  4. 2026MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
  5. 2024Large Language Models Can Self-Improve At Web Agent Tasks
  6. 2026Avenir-Web: Human-Experience-Imitating Multimodal Web Agents with Mixture of Grounding Experts
  7. 2025Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
  8. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code