mlsanityData to 7 Oct 2026

Harnessing Webpage UIs for Text-Rich Visual Understanding

arXiv 2410.13824 ↗First posted 17 Oct 2024cs.CV cs.CLcited by 28

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Text-rich visual understanding-the ability to process environments where dense textual content is integrated with visuals-is crucial for multimodal large language models (MLLMs) to interact effectively with structured environments. To enhance this capability, we propose synthesizing general multimodal instructions from webpage UIs using text-based large language models (LLMs). Despite lacking direct visual input, text-based LLMs are able to process structured text representations from webpage accessibility trees. These instructions are then paired with UI screenshots to train multimodal models. We introduce MultiUI, a dataset containing 7.3 million samples from 1 million websites, covering diverse multimodal tasks and UI layouts. Models trained on MultiUI not only excel in web UI tasks-achieving up to a 48% improvement on VisualWebBench and a 19.1% boost in element accuracy on a web agent dataset Mind2Web-but also generalize surprisingly well to non-web UI tasks and even to non-UI domains, such as document understanding, OCR, and chart interpretation. These results highlight the broad applicability of web UI data for advancing text-rich visual understanding across various scenarios.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 50 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

38 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?9 Apr

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
25 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2016Modeling Context in Referring Expressionscited by 1,054
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  7. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  8. 2021InfographicVQAcited by 430
  9. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  10. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  11. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  13. 2019Character Region Awareness for Text Detectioncited by 273
  14. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  15. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  16. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  17. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  18. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  19. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  20. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  21. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  22. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  23. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  24. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  25. 2018Resolving Referring Expressions in Images With Labeled Elementscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 8.

28 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  4. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  5. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  8. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  9. 2025Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Modelscited by 24
  10. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  11. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  12. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
16 more
  1. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 19
  2. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  3. 2025R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memorycited by 14
  4. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalizationcited by 13
  5. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curationcited by 13
  6. 2026WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agentscited by 6
  7. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  8. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agentscited by 3
  9. 2025Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agentscited by 2
  10. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  11. 2026ZAYA1-VL-8B Technical Reportcited by 2
  12. 2026Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approachcited by 1
  13. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 1
  14. 2026DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documentscited by 1
  15. 2025GUIrilla: A Scalable Framework for Automated Desktop UI Explorationcited by 0
  16. 2026Zamba2-VL Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
  2. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation
  3. 2026MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  5. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
  6. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code