Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
GUI agents increasingly operate across websites, mobile apps, and desktop environments, yet the field still reports progress primarily through task success. We argue that practical deployment depends equally on efficiency: how much context, computation, action budget, and runtime overhead an agent consumes while succeeding. This survey studies efficient GUI agents through an end-to-end systems lens that preserves the current technical axes of observation efficiency, context and memory efficiency, action efficiency, and planner-side/system efficiency. For each subsection, we expand the seed literature through targeted search plus backward and forward citation chaining, then synthesize the dominant mechanisms, reported efficiency signals, and new overheads they introduce. Across the literature, recent progress converges on a small set of recurring ideas: selective reading instead of full-context ingestion, global-to-local visual allocation, recoverable memory rather than raw history replay, verification-aware control, and hybrid runtimes that can switch between GUI and non-GUI execution. We conclude by identifying the main open problems, including honest accounting of verifier cost, cross-benchmark comparability, and co-design of observation, memory, and execution layers under real latency and privacy constraints.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 59 of its 75 references that are in the map. First posted on arXiv 2 Sep 2026.
Builds on
59 of 75 references.
Closest ancestors · 25
Same direction, newest first
- 2026SecAgent: Efficient Mobile GUI Agent with Semantic Context9 Mar
- 2025HiconAgent: History Context-aware Policy Optimization for GUI Agents1 Dec
- 2025Auto-scaling Continuous Memory for GUI Agent10 Oct
- 2025GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness1 Oct
- 2025PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents1 Oct
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents25 Jul
- 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents19 Jun
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
- 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism27 May
- 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent22 May
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents21 May
- 2025Visual Test-time Scaling for GUI Agent Grounding1 May
13 more
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification24 Mar
- 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment20 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
Other references · 34
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
- 2024OmniParser for Pure Vision Based GUI Agentcited by 198
- 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
- 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
- 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
- 2024Efficient Multimodal Large Language Models: A Surveycited by 96
- 2024Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systemscited by 81
22 more
- 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
- 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
- 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 46
- 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
- 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
- 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
- 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Groundingcited by 22
- 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 21
- 2024Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Groundingcited by 18
- 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
- 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
- 2024Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agentscited by 13
- 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 12
- 2025WebOperator: Action-Aware Tree Search for Autonomous Agents in Web Environmentcited by 10
- 2026ColorBrowserAgent: Complex Long-Horizon Browser Agent with Adaptive Knowledge Evolutioncited by 6
- 2025LineRetriever: Planning-Aware Observation Reduction for Web Agentscited by 5
- 2025Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agentscited by 4
- 2025GUI-Rise: Structured Reasoning and History Summarization for GUI Navigationcited by 4
- 2026Read More, Think More: Revisiting Observation Reduction for Web Agentscited by 4
- 2025Local Information Matters: Inference Acceleration For Grounded Conversation Generation Models Through Adaptive Local-Aware Token Pruningcited by 2
- 2026IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agentscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms
- 2026Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
- 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
- 2026ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory
- 2026JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
- 2026DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents
- 2026A Task-State Representation for Long-Horizon Mobile GUI Agents