mlsanityData to 14 Oct 2026

Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

arXiv 2010.04295 ↗First posted 8 Oct 2020cs.LG cs.AI cs.CLcited by 95

Not on the map. Why?

Cited by 31 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general. Yet, these descriptions are constantly missing in mobile UIs. We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces. We collected a large-scale dataset for widget captioning with crowdsourcing. Our dataset contains 162,859 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens. We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions. The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 9; 2023: 10; 2024: 24; 2025: 32; 2026: 15.

95 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  7. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  9. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  10. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  11. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  12. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
38 more
  1. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  2. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  3. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  4. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  5. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
  6. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  7. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  8. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
  9. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  10. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  11. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  12. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
  13. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  14. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 68
  15. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  16. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  17. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  18. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  19. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 55
  20. 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
  21. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  22. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  23. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  24. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  25. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  26. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  27. 2022UGIF: UI Grounded Instruction Followingcited by 25
  28. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  29. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  30. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 23
  31. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  32. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  33. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
  34. 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
  35. 2025UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesiscited by 20
  36. 2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 19
  37. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  38. 2023Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation APIcited by 18

50 most cited of 95 All 95 (JSON)