mlsanityData to 7 Oct 2026

You Only Look at Screens: Multimodal Chain-of-Action Agents

arXiv 2309.11436 ↗First posted 20 Sep 2023cs.CL cs.AI cs.HCcited by 185

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Autonomous graphical user interface (GUI) agents aim to facilitate task automation by interacting with the user interface without manual intervention. Recent studies have investigated eliciting the capabilities of large language models (LLMs) for effective engagement in diverse environments. To align with the input-output requirement of LLMs, most existing approaches are developed under a sandbox setting where they rely on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. Consequently, those approaches often grapple with inference inefficiency and error propagation risks. To mitigate the challenges, we introduce Auto-GUI, a multimodal solution that directly interacts with the interface, bypassing the need for environment parsing or reliance on application-dependent APIs. Moreover, we propose a chain-of-action technique -- leveraging a series of intermediate previous action histories and future action plans -- to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30K unique instructions, spanning multi-step tasks such as application operation, web searching, and web shopping. Experimental results show that Auto-GUI achieves state-of-the-art performance with an action type prediction accuracy of 90% and an overall action success rate of 74%. Code is publicly available at https://github.com/cooelf/Auto-GUI.

Key facts. Cited by 185 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 58 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

31 of 58 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
19 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  8. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  9. 2023Cognitive Architectures for Language Agentscited by 422
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  12. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  13. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  14. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  15. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  16. 2021Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translationcited by 44
  17. 2022On Vision Features in Multimodal Machine Translationcited by 31
  18. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  19. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 57; 2025: 81; 2026: 39.

185 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  5. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  6. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  9. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  10. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  11. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
38 more
  1. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  2. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  3. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  5. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  6. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  7. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  8. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  9. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  10. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  11. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  12. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
  13. 2024MMedAgent: Learning to Use Medical Tools with Multi-modal Agentcited by 108
  14. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  15. 2024Large Multimodal Agents: A Surveycited by 99
  16. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 93
  17. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  18. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  19. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  20. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  21. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  22. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  23. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  24. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  25. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  26. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  27. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  28. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  29. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54
  30. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  31. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  32. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  33. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  34. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  35. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  36. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  37. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  38. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37

50 most cited of 185 All 185 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025API Agents vs. GUI Agents: Divergence and Convergence
  2. 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
  3. 2026Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization
  4. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  5. 2026UI-Venus-2 Technical Report
  6. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
  7. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents
  8. 2024GUI Agents: A Survey