mlsanityData to 7 Oct 2026

WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

arXiv 2301.13280 ↗First posted 30 Jan 2023cs.HCcited by 45

Not on the map. Why?

Abstract (arXiv)

Modeling user interfaces (UIs) from visual information allows systems to make inferences about the functionality and semantics needed to support use cases in accessibility, app automation, and testing. Current datasets for training machine learning models are limited in size due to the costly and time-consuming process of manually collecting and annotating UIs. We crawled the web to construct WebUI, a large dataset of 400,000 rendered web pages associated with automatically extracted metadata. We analyze the composition of WebUI and show that while automatically extracted data is noisy, most examples meet basic criteria for visual UI modeling. We applied several strategies for incorporating semantics found in web pages to increase the performance of visual UI understanding models in the mobile domain, where less labeled data is available: (i) element detection, (ii) screen classification and (iii) screen similarity.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jan 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 19; 2026: 7.

45 papers.

Sort:

By area:

  1. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  4. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  5. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  6. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  7. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  8. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  9. 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
  10. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  11. 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 26
  12. 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
33 more
  1. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  2. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  3. 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
  4. 2024Automating the Enterprise with Foundation Modelscited by 14
  5. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  6. 2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
  7. 2025PICD: Versatile Perceptual Image Compression with Diffusion Renderingcited by 10
  8. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  9. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  10. 2024Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refiningcited by 7
  11. 2025LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generationcited by 6
  12. 2025Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environmentscited by 6
  13. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 6
  14. 2025STEVE: A Step Verification Pipeline for Computer-use Agent Trainingcited by 5
  15. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  16. 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 5
  17. 2023MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learningcited by 4
  18. 2024The Impact of Element Ordering on LM Agent Performancecited by 3
  19. 2025PixelWeb: The First Web GUI Dataset with Pixel-Wise Labelscited by 3
  20. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agentscited by 3
  21. 2026POINTS-GUI-G: GUI-Grounding Journeycited by 3
  22. 2025Explorer: Robust Collection of Interactable GUI Elementscited by 2
  23. 2025Multimodal graph representation learning for website generation based on visual sketchcited by 2
  24. 2024Automatically Detecting Online Deceptive Patternscited by 1
  25. 2025Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UIcited by 1
  26. 2026UXBench: Measuring the Actionability of LLM-Generated UX Critiquescited by 1
  27. 2024Computer User Interface Understanding. A New Dataset and a Learning Frameworkcited by 0
  28. 2025ControlGUI: Guiding Generative GUI Exploration through Perceptual Visual Flowcited by 0
  29. 2025RWKV-UI: UI Understanding with Enhanced Perception and Reasoningcited by 0
  30. 2026A Lightweight Vision-Language Fusion Framework for Predicting App Ratings from User Interfaces and Metadatacited by 0
  31. 2026HiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledgecited by 0
  32. 2026Efficient Personalization of Generative User Interfacescited by 0
  33. 2026GoClick: Lightweight Element Grounding Model for Autonomous GUI Interactioncited by 0