mlsanityData to 7 Oct 2026

VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?

arXiv 2404.05955 ↗First posted 9 Apr 2024cs.CL cs.AIcited by 98

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce bench, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. bench consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites, covering 87 sub-domains. We evaluate 14 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on bench, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe bench will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 44 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

35 of 44 references.

References · 35

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
23 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  5. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2024Yi: Open Foundation Models by 01.AIcited by 706
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  13. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  14. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  15. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  16. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  18. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  19. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  20. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  21. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  22. 2024Dual-View Visual Contextualization for Web Navigationcited by 24
  23. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 49; 2026: 20.

98 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  3. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  4. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  5. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  6. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  7. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  10. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  11. 2024Tree Search for Language Model Agentscited by 148
  12. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
38 more
  1. 2025MiMo-VL Technical Reportcited by 106
  2. 2024GUI Agents: A Surveycited by 101
  3. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  5. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  6. 2024Beyond Browsing: API-Based Web Agentscited by 77
  7. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  8. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  9. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  10. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  11. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  12. 2025Step-GUI Technical Reportcited by 38
  13. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  14. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  15. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  16. 2026MiniMax Sparse Attentioncited by 34
  17. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  18. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  19. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  20. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  21. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  22. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  23. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  24. 2025LongCat-Flash-Omni Technical Reportcited by 23
  25. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  26. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  27. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
  28. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 17
  29. 2025Build the web for agents, not agents for the webcited by 17
  30. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  31. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  32. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  33. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  34. 2024WebQuest: A Benchmark for Multimodal QA on Web Page Sequencescited by 12
  35. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  36. 2024ClickAgent: Enhancing UI Location Capabilities of Autonomous Agentscited by 11
  37. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  38. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Datacited by 10

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
  2. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMs
  3. 2025Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety
  4. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code
  5. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  6. 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
  7. 2024MileBench: Benchmarking MLLMs in Long Context
  8. 2024The Revolution of Multimodal Large Language Models: A Survey