mlsanityData to 7 Oct 2026

Mapping Natural Language Instructions to Mobile UI Action Sequences

arXiv 2005.03776 ↗First posted 7 May 2020cs.CL cs.LGcited by 180

Not on the map. Why?

Cited by 77 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it. For full task evaluation, we create PIXELHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator. To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces. We use a Transformer to extract action phrase tuples from long-range natural language instructions. A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions. Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PIXELHELP.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 12; 2023: 24; 2024: 52; 2025: 58; 2026: 18.

180 papers.

Sort:

By area:

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  7. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  8. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  9. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  12. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
38 more
  1. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  2. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  3. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  4. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  5. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  6. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  7. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  8. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  9. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  10. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  11. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  12. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  13. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  14. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  15. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  16. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  17. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  18. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  19. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  20. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  21. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  22. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  23. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  24. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  25. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  26. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  27. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  28. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 93
  29. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  30. 2022A data-driven approach for learning to control computerscited by 87
  31. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  32. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  33. 2024Beyond Browsing: API-Based Web Agentscited by 77
  34. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  35. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  36. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  37. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  38. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65

50 most cited of 180 All 180 (JSON)