Data to 14 Oct 2026

Large Language Model-Brained GUI Agents: A Survey

arXiv 2411.18279 ↗First posted 27 Nov 2024cs.AI cs.CL cs.HCcited by 162

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

GUIs have long been central to human-computer interaction, providing an intuitive and visually-driven way to access and interact with digital systems. The advent of LLMs, particularly multimodal models, has ushered in a new era of GUI automation. They have demonstrated exceptional capabilities in natural language understanding, code generation, and visual processing. This has paved the way for a new generation of LLM-brained GUI agents capable of interpreting complex GUI elements and autonomously executing actions based on natural language instructions. These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands. Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software. This emerging field is rapidly advancing, with significant progress in both research and industry. To provide a structured understanding of this trend, this paper presents a comprehensive survey of LLM-brained GUI agents, exploring their historical evolution, core components, and advanced techniques. We address research questions such as existing GUI agent frameworks, the collection and utilization of data for training specialized GUI agents, the development of large action models tailored for GUI tasks, and the evaluation metrics and benchmarks necessary to assess their effectiveness. Additionally, we examine emerging applications powered by these agents. Through a detailed analysis, this survey identifies key research gaps and outlines a roadmap for future advancements in the field. By consolidating foundational knowledge and state-of-the-art developments, this work aims to guide both researchers and practitioners in overcoming challenges and unlocking the full potential of LLM-brained GUI agents.

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 283 of its 636 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

283 of 636 references.

Closest ancestors · 42

Same direction, newest first

30 more

Other references · 241

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2020Language Models are Few-Shot Learnerscited by 32,419
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2023GPT-4 Technical Reportcited by 18,396
2024The Llama 3 Herd of Modelscited by 14,715
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
2021Evaluating Large Language Models Trained on Codecited by 8,320
2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
2024GPT-4o System Cardcited by 4,588

38 more

2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2023Qwen Technical Reportcited by 3,344
2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
2023Mistral 7Bcited by 3,164
2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
2023A Survey of Large Language Modelscited by 2,576
2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
2023Code Llama: Open Foundation Models for Codecited by 2,139
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
2023AgentBench: Evaluating LLMs as Agentscited by 1,267
2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
2023PaLM 2 Technical Reportcited by 1,162
2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
2023A Survey on Multimodal Large Language Modelscited by 909
2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
2022A Survey on In-context Learningcited by 721
2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
2024PaliGemma: A versatile 3B VLM for transfercited by 696
2023A Comprehensive Overview of Large Language Modelscited by 648
2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
2023Instruction Tuning for Large Language Models: A Surveycited by 623
2022Towards Reasoning in Large Language Models: A Surveycited by 596
2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438

50 most cited of 241 All 241 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 102; 2026: 58.

162 papers.

Sort:

By area:

2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 195
2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 186
2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
2024GUI Agents: A Surveycited by 103
2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95

38 more

2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
2025Latent Collaboration in Multi-Agent Systemscited by 73
2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 63
2025A Survey on (M)LLM-Based GUI Agentscited by 62
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 49
2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 46
2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 42
2025UFO2: The Desktop AgentOScited by 41
2025ViMo: A Generative Visual GUI World Model for App Agentscited by 35
2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 34
2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 34
2025Towards Trustworthy GUI Agents: A Surveycited by 34
2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 33
2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Toolscited by 32
2025API Agents vs. GUI Agents: Divergence and Convergencecited by 29
2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
2026TimeART: Towards Agentic Time Series Reasoning via Tool-Augmentationcited by 26
2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 25
2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 21
2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 20
2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
2025AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applicationscited by 17
2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 13
2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling Systemcited by 12
2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way