mlsanityData to 7 Oct 2026

GUI Agents with Foundation Models: A Comprehensive Survey

arXiv 2411.04890 ↗First posted 7 Nov 2024cs.AI cs.HCcited by 110

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), have facilitated the development of intelligent agents capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions, simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data resources, frameworks, and applications. We begin by reviewing representative datasets and benchmarks, followed by an overview of a generalized, unified framework that encapsulates the essential components of prior studies, supported by a detailed taxonomy. Additionally, we explore relevant commercial applications. Drawing insights from existing work, we identify key challenges and propose future research directions. We hope this survey will inspire further advancements in the field of (M)LLM-based GUI agents.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 44 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

30 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  2. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov
  3. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations7 Oct
  4. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus29 Sep

Other references · 26

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  6. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  7. 2023Language Models can Solve Computer Taskscited by 371
  8. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  9. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  10. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  11. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  12. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
14 more
  1. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  2. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  3. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  4. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  5. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  6. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  7. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  8. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  9. 2022A data-driven approach for learning to control computerscited by 87
  10. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 79
  11. 2018Learning to Navigate the Webcited by 60
  12. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  13. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  14. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 68; 2026: 41.

110 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  5. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  6. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  7. 2025Agentic Large Language Models, a surveycited by 105
  8. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  9. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  10. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  11. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  12. 2025Training-Free Group Relative Policy Optimizationcited by 53
38 more
  1. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  2. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  3. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  4. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  5. 2026Code as Agent Harnesscited by 40
  6. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  7. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  8. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  9. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
  10. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  11. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  12. 2025LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarioscited by 25
  13. 2025VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verificationcited by 20
  14. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  15. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  16. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  17. 2026GUI Agents for Continual Game Generationcited by 16
  18. 2025Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinementcited by 15
  19. 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 15
  20. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  21. 2025ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Searchcited by 12
  22. 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 12
  23. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 11
  24. 2025Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agentscited by 11
  25. 2025PAL-UI: Planning with Active Look-back for Vision-Based GUI Agentscited by 11
  26. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  27. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
  28. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Spacecited by 10
  29. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 10
  30. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  31. 2025COLA: A Scalable Multi-Agent Framework For Windows UI Task Automationcited by 9
  32. 2025GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agentscited by 8
  33. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environmentscited by 8
  34. 2025Scaling Laws for Code: Every Programming Language Matterscited by 8
  35. 2025Characterizing Unintended Consequences in Human-GUI Agent Collaboration for Web Browsingcited by 7
  36. 2025DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agentscited by 7
  37. 2026ShowUI-Aloha: Human-Taught GUI Agentcited by 7
  38. 2025CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planningcited by 6

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025API Agents vs. GUI Agents: Divergence and Convergence
  2. 2024GUI Agents: A Survey
  3. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  4. 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
  5. 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
  6. 2025From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents
  7. 2024Large Language User Interfaces: Voice Interactive User Interfaces powered by LLMs
  8. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding