mlsanityData to 7 Oct 2026

Agent S: An Open Agentic Framework that Uses Computers Like a Human

arXiv 2410.08164 ↗First posted 10 Oct 2024cs.AI cs.CL cs.CVcited by 182

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

We present Agent S, an open agentic framework that enables autonomous interaction with computers through a Graphical User Interface (GUI), aimed at transforming human-computer interaction by automating complex, multi-step tasks. Agent S aims to address three key challenges in automating computer tasks: acquiring domain-specific knowledge, planning over long task horizons, and handling dynamic, non-uniform interfaces. To this end, Agent S introduces experience-augmented hierarchical planning, which learns from external knowledge search and internal experience retrieval at multiple levels, facilitating efficient task planning and subtask execution. In addition, it employs an Agent-Computer Interface (ACI) to better elicit the reasoning and control capabilities of GUI agents based on Multimodal Large Language Models (MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves a new state-of-the-art. Comprehensive analysis highlights the effectiveness of individual components and provides insights for future improvements. Furthermore, Agent S demonstrates broad generalizability to different operating systems on a newly-released WindowsAgentArena benchmark. Code available at https://github.com/simular-ai/Agent-S.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 40 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

31 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  3. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  10. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
16 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  2. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  4. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  5. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  7. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  8. 2023Cognitive Architectures for Language Agentscited by 422
  9. 2023Language Models can Solve Computer Taskscited by 371
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  12. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  13. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  14. 2022A data-driven approach for learning to control computerscited by 87
  15. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agentscited by 80
  16. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 90; 2026: 88.

182 papers.

Sort:

By area:

  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  2. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  5. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  6. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  7. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  10. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  11. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  12. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
38 more
  1. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  2. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  3. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  5. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  6. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 46
  7. 2025PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PCcited by 39
  8. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  9. 2026UI-Venus-1.5 Technical Reportcited by 39
  10. 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
  11. 2025UFO2: The Desktop AgentOScited by 38
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  13. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  14. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 34
  15. 2025Towards Trustworthy GUI Agents: A Surveycited by 33
  16. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 33
  17. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  18. 2025Supervising the search process produces reliable and generalizable information-seeking agentscited by 20
  19. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 20
  20. 2024Infogent: An Agent-Based Framework for Web Information Aggregationcited by 19
  21. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 19
  22. 2025Scaling Synthetic Task Generation for Agents via Explorationcited by 19
  23. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  24. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  25. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  26. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 18
  27. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  28. 2025Efficient Agent Training for Computer Usecited by 17
  29. 2026Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agentscited by 17
  30. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 16
  31. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 15
  32. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 15
  33. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 15
  34. 2025Dyna-Think: Synergizing Reasoning, Acting, and World Model Simulation in AI Agentscited by 14
  35. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 14
  36. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  37. 2025Scaling Agents for Computer Usecited by 14
  38. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 13

50 most cited of 182 All 182 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents
  2. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  3. 2024GUI Agents: A Survey
  4. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
  5. 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
  6. 2026KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
  7. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents
  8. 2026CUA-Skill: Develop Skills for Computer Using Agent