mlsanityData to 7 Oct 2026

GPT-4V(ision) is a Generalist Web Agent, if Grounded

arXiv 2401.01614 ↗First posted 3 Jan 2024cs.IR cs.AI cs.CLcited by 547

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. We evaluate on the recent MIND2WEB benchmark. In addition to standard offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents -- it can successfully complete 51.1 of the tasks on live websites if we manually ground its textual plans into actions on the websites. This substantially outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2) specifically fine-tuned for web agents. However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out to be not effective for web agents, and the best grounding strategy we develop in this paper leverages both the HTML structure and visuals. Yet, there is still a substantial gap with oracle grounding, leaving ample room for further improvement. All code, data, and evaluation tools are available at https://github.com/OSU-NLP-Group/SeeAct.

Key facts. Cited by 547 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 67 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

55 of 67 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  15. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  16. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  17. 2023Instruction Tuning with GPT-4cited by 676
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  19. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  20. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  21. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  22. 2023Language Models can Solve Computer Taskscited by 371
  23. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  24. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  25. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  26. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  27. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  28. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  29. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  30. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  31. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  32. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  33. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  34. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  35. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  36. 2023Fine-Grained Visual Promptingcited by 81
  37. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  38. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 128; 2025: 216; 2026: 203.

547 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  5. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  7. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  8. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  9. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  10. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  11. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  12. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
38 more
  1. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  2. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  3. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  4. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  5. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  6. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  7. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  8. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  9. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  10. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  11. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  12. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  13. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  14. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  15. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  16. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  17. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  18. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  19. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  20. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  21. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  22. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  23. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  24. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  25. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  26. 2024Tree Search for Language Model Agentscited by 148
  27. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  28. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  29. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  30. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  31. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  32. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  33. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  34. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  35. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  36. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  37. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  38. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123

50 most cited of 547 All 547 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO
  2. 2026WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites
  3. 2026Improving LLM-based Autonomous Web Agents with Filtering
  4. 2024VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework
  5. 2024GTA: A Benchmark for General Tool Agents
  6. 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
  7. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn
  8. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent