mlsanityData to 7 Oct 2026

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

arXiv 2409.08264 ↗First posted 12 Sep 2024cs.AIcited by 212

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena. Webpage: https://microsoft.github.io/WindowsAgentArena Code: https://github.com/microsoft/WindowsAgentArena

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 39 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

20 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  2. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  3. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  8. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  9. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  10. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  11. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
  12. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
5 more
  1. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  2. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  3. 2022A data-driven approach for learning to control computerscited by 87
  4. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  5. 2021Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environmentscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 86; 2026: 117.

212 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2025Kimi-VL Technical Reportcited by 381
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  9. 2025Survey on Evaluation of LLM-based Agentscited by 190
  10. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  11. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  12. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
38 more
  1. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  2. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  3. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  4. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 126
  5. 2024GUI Agents: A Surveycited by 101
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  7. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  8. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  9. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  10. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
  11. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  12. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  13. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  14. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  15. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  16. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  17. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 54
  18. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  19. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  20. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  21. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  22. 2026Code as Agent Harnesscited by 40
  23. 2025PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PCcited by 39
  24. 2025UFO2: The Desktop AgentOScited by 38
  25. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  26. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  27. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  28. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  29. 2025A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directionscited by 33
  30. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 32
  31. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  32. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  33. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 30
  34. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 29
  35. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agentscited by 25
  36. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 19
  37. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  38. 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 19

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
  2. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  3. 2023AgentBench: Evaluating LLMs as Agents
  4. 2026OSWorld-Pro: Process-based Evaluation for Computer Use Agents
  5. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
  6. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human
  7. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  8. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution