mlsanityData to 14 Oct 2026

From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces

arXiv 2306.00245 ↗First posted 31 May 2023cs.LG cs.CL cs.CVcited by 68

Not on the map. Why?

Cited by 44 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use -- via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

22 of 40 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  11. 2023Language Models can Solve Computer Taskscited by 372
  12. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
10 more
  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  2. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  3. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  4. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  5. 2023Vision-Language Models as Success Detectorscited by 142
  6. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
  7. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
  8. 2022A data-driven approach for learning to control computerscited by 88
  9. 2018Learning to Navigate the Webcited by 60
  10. 2019DOM-Q-NET: Grounded RL on Structured Languagecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 27; 2025: 23; 2026: 9.

68 papers.

Sort:

By area:

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  7. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  8. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  9. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  10. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  12. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
38 more
  1. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  2. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
  3. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  5. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 161
  6. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  7. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  8. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  9. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  10. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
  11. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  12. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
  13. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  14. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  15. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 55
  16. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 46
  17. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  18. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  19. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  20. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  21. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  22. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  23. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  24. 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
  25. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 31
  26. 2023A Zero-Shot Language Agent for Computer Control with Structured Reflectioncited by 28
  27. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  28. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  29. 2023Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Webcited by 24
  30. 2024Dual-View Visual Contextualization for Web Navigationcited by 24
  31. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 19
  32. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  33. 2025Build the web for agents, not agents for the webcited by 18
  34. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  35. 2025On the Robustness of GUI Grounding Models Against Image Attackscited by 14
  36. 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 11
  37. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  38. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9

50 most cited of 68 All 68 (JSON)