mlsanityData to 7 Oct 2026

GUI Agents: A Survey

arXiv 2412.13501 ↗First posted 18 Dec 2024cs.AI cs.HCcited by 101

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 71 of its 102 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

71 of 102 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  4. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  5. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions5 Aug
  6. 2024MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices4 Jul
  7. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  8. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  9. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  10. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  11. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  12. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
38 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  4. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  5. 2024Understanding the planning of LLM agents: A surveycited by 414
  6. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  7. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  8. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  9. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  10. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  11. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  13. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  14. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  15. 2024Tree Search for Language Model Agentscited by 148
  16. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  17. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  18. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  19. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  20. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  21. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  22. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
  23. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  24. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  25. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  26. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  27. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 81
  28. 2024Beyond Browsing: API-Based Web Agentscited by 77
  29. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  30. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  31. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  32. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  33. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
  34. 2024On the Multi-turn Instruction Following for Conversational Web Agentscited by 42
  35. 2022Plansformer: Generating Symbolic Plans using Transformerscited by 41
  36. 2023Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisionscited by 41
  37. 2022Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testingcited by 36
  38. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 30

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 44; 2026: 57.

101 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  4. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  5. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  7. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  8. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  9. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  10. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  11. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  12. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
38 more
  1. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  2. 2025Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attackscited by 21
  3. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  4. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  5. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  6. 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomaliescited by 15
  7. 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 15
  8. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  9. 2025Mano Technical Reportcited by 13
  10. 2026Generative Visual Code Mobile World Modelscited by 13
  11. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 12
  12. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  13. 2025Are Your Agents Upward Deceivers?cited by 11
  14. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  15. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 10
  16. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
  17. 2026OpenComputer: Verifiable Software Worlds for Computer-Use Agentscited by 9
  18. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  19. 2025GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agentscited by 8
  20. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  21. 2025Understanding GUI Agent Localization Biases through Logit Sharpnesscited by 7
  22. 2025DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agentscited by 7
  23. 2026ShowUI-Aloha: Human-Taught GUI Agentcited by 7
  24. 2026Step-level Optimization for Efficient Computer-use Agentscited by 7
  25. 2025CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planningcited by 6
  26. 2025LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agentscited by 6
  27. 2025CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learningcited by 6
  28. 2025PG-Agent: An Agent Powered by Page Graphcited by 6
  29. 2025OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Modelscited by 6
  30. 2025Mobile GUI Agents under Real-world Threats: Are We There Yet?cited by 5
  31. 2025DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfercited by 5
  32. 2025Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environmentscited by 5
  33. 2025BTL-UI: Blink-Think-Link Reasoning Model for GUI Agentcited by 5
  34. 2025PhysiAgent: An Embodied Agent Framework in Physical Worldcited by 5
  35. 2026Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisiblecited by 5
  36. 2025MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Modelscited by 4
  37. 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulationcited by 4
  38. 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GUI Agents with Foundation Models: A Comprehensive Survey
  2. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  3. 2025Computer-Use Agents as Judges for Generative User Interface
  4. 2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
  5. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents
  6. 2024TinyClick: Single-Turn Agent for Empowering GUI Automation
  7. 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
  8. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction