mlsanityData to 7 Oct 2026

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

arXiv 2404.07972 ↗First posted 11 Apr 2024cs.AI cs.CLcited by 1,134

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.

Key facts. Cited by 1,134 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 48 of its 63 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

48 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  2. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  3. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  5. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  6. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction14 May

Other references · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
30 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  4. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  5. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  6. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  7. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  8. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  9. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  10. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  11. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  12. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  13. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  14. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  15. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  16. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  17. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  18. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  19. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  20. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  21. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  22. 2022META-GUI: Towards Multi-modal Conversational Agents on Mobile GUIcited by 81
  23. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  24. 2023SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Modelscited by 53
  25. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automationcited by 45
  26. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  27. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
  28. 2023Hierarchical Prompting Assists Large Language Model on Web Navigationcited by 36
  29. 2023PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completioncited by 30
  30. 2022UGIF: UI Grounded Instruction Followingcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 65; 2025: 288; 2026: 781.

1,134 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  4. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  5. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  6. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  7. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  8. 2025Kimi-VL Technical Reportcited by 381
  9. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  10. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  11. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  12. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
38 more
  1. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  2. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  3. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  4. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  5. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
  6. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  7. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  8. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  10. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  11. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  13. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  14. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  15. 2025Survey on Evaluation of LLM-based Agentscited by 190
  16. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  17. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  18. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  19. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  20. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  21. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  22. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  23. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  24. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  25. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  26. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  27. 2024Tree Search for Language Model Agentscited by 148
  28. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  29. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  30. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  31. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  32. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  33. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 119
  34. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  35. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  36. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  37. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 114
  38. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112

50 most cited of 1,134 All 1,134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
  2. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
  3. 2025Scaling Synthetic Task Generation for Agents via Exploration
  4. 2024MMInA: Benchmarking Multihop Multimodal Internet Agents
  5. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
  6. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  7. 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
  8. 2026CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents