mlsanityData to 7 Oct 2026

Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

arXiv 2504.00906 ↗First posted 1 Apr 2025cs.AI cs.CL cs.CVcited by 156

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open-ended space of user queries. However, current agents face significant challenges: imprecise grounding of GUI elements, difficulties with long-horizon task planning, and performance bottlenecks from relying on single generalist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novel compositional framework that delegates cognitive responsibilities across various generalist and specialist models. We propose a novel Mixture-of-Grounding technique to achieve precise GUI localization and introduce Proactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evaluations demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance on three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such as Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentArena and by 16.52% on AndroidWorld relatively. Code available at https://github.com/simular-ai/Agent-S.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 27 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

22 of 27 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  3. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  5. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  6. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions5 Aug
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  8. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  9. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 13

  1. 2014How transferable are features in deep neural networks?cited by 1,216
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  4. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  8. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  9. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  10. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 93
  11. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  12. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
1 more
  1. 2024AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistantcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 67; 2026: 89.

156 papers.

Sort:

By area:

  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  3. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  4. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  5. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 105
  6. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  7. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  8. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  9. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  10. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  11. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  12. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
38 more
  1. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  2. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 41
  3. 2026UI-Venus-1.5 Technical Reportcited by 39
  4. 2025Step-GUI Technical Reportcited by 38
  5. 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agentscited by 37
  6. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  7. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 33
  8. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  9. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 30
  10. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  11. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
  12. 2025PolySkill: Learning Generalizable Skills Through Polymorphic Abstractioncited by 25
  13. 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
  14. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  15. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  16. 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 18
  17. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  18. 2025FocusAgent: Simple Yet Effective Ways of Trimming the Large Context of Web Agentscited by 16
  19. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 16
  20. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
  21. 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 15
  22. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 15
  23. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 15
  24. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  25. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  26. 2025Scaling Agents for Computer Usecited by 14
  27. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 12
  28. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  29. 2026The Art of Building Verifiers for Computer Use Agentscited by 11
  30. 2025BIMgent: Towards Autonomous Building Modeling via Computer-use Agentscited by 10
  31. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  32. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
  33. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 10
  34. 2025AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Deliverycited by 9
  35. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operationcited by 9
  36. 2025Watch and Learn: Learning to Use Computers from Online Videoscited by 9
  37. 2026Recursive Models for Long-Horizon Reasoningcited by 9
  38. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
  2. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  3. 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
  4. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
  5. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  6. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
  7. 2025Computer-Use Agents as Judges for Generative User Interface
  8. 2025COLA: A Scalable Multi-Agent Framework For Windows UI Task Automation