mlsanityData to 7 Oct 2026

OS-Copilot: Towards Generalist Computer Agents with Self-Improvement

arXiv 2402.07456 ↗First posted 12 Feb 2024cs.AIcited by 203

On the map: LLM agents and tool use → GUI and mobile agents

also in 1 more
Abstract (arXiv)

Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 50 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

40 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  2. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  3. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 37

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
25 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  5. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2023Cognitive Architectures for Language Agentscited by 422
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  10. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  11. 2023Tool Learning with Foundation Modelscited by 302
  12. 2023Large Language Models as Tool Makerscited by 293
  13. 2023AI Alignment: A Comprehensive Surveycited by 256
  14. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  15. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  16. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  17. 2023TaskWeaver: A Code-First Agent Frameworkcited by 85
  18. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  19. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  20. 2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 53
  21. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  22. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  23. 2023PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completioncited by 30
  24. 2022Unsupervised Explanation Generation via Correct Instantiationscited by 5
  25. 2022Explanation Regeneration via Information Bottleneckcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 94; 2026: 56.

203 papers.

Sort:

By area:

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  4. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  5. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  7. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  8. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  9. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  10. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  11. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  12. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
38 more
  1. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  2. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  5. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  6. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  7. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  10. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  11. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  12. 2024Large Multimodal Agents: A Surveycited by 99
  13. 2024AIOS: LLM Agent Operating Systemcited by 94
  14. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 90
  15. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  16. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  17. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 80
  18. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  19. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  20. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  21. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  22. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  23. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  24. 2025Scaling Test-time Compute for LLM Agentscited by 59
  25. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  26. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  27. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  28. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  29. 2024TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generationcited by 50
  30. 2025AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agentscited by 49
  31. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  32. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  33. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  34. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  35. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  36. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  37. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  38. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38

50 most cited of 203 All 203 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
  2. 2024From Language Models to Practical Self-Improving Computer Agents
  3. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
  4. 2026Computer Environments Elicit General Agentic Intelligence in LLMs
  5. 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios
  6. 2023Language Models can Solve Computer Tasks
  7. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
  8. 2026OSExpert: Computer-Use Agents Learning Professional Skills via Exploration