# OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents

arXiv: https://arxiv.org/abs/2506.16042 · HTML: https://mlsanity.com/paper/2506.16042/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [Infant Agent: A Tool-Integrated, Logic-Driven Agent with Cost-Effective API Usage](https://mlsanity.com/paper/2411.01114/) (2024)

## Built on by (most cited first, 30 of 38)

- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents](https://mlsanity.com/paper/2510.24563/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [Speculative Actions: A Lossless Framework for Faster Agentic Systems](https://mlsanity.com/paper/2510.04371/) (2025)
- [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026)
- [AgentAtlas: Beyond Outcome Leaderboards for LLM Agents](https://mlsanity.com/paper/2605.20530/) (2026)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environments](https://mlsanity.com/paper/2601.02399/) (2025)
- [OSExpert: Computer-Use Agents Learning Professional Skills via Exploration](https://mlsanity.com/paper/2603.07978/) (2026)
- [Step-level Optimization for Efficient Computer-use Agents](https://mlsanity.com/paper/2604.27151/) (2026)
- [WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents](https://mlsanity.com/paper/2603.05044/) (2026)
- [Learning Latency-Aware Orchestration for Multi-Agent Systems](https://mlsanity.com/paper/2601.10560/) (2026)
- [LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis](https://mlsanity.com/paper/2605.30434/) (2026)
- [UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability](https://mlsanity.com/paper/2511.17131/) (2025)
- [ANCHOR: Branch-Point Data Generation for GUI Agents](https://mlsanity.com/paper/2602.07153/) (2026)
- [Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents](https://mlsanity.com/paper/2605.02729/) (2026)
- [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](https://mlsanity.com/paper/2605.17170/) (2026)
- [Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents](https://mlsanity.com/paper/2608.03327/) (2026)
- [OS-Marathon: Benchmarking Computer-Use Agents on Vast-Horizon, Repetitive Tasks](https://mlsanity.com/paper/2601.20650/) (2026)
- [PANDO: Efficient Multimodal AI Agents via Online Skill Distillation](https://mlsanity.com/paper/2605.24785/) (2026)
- [Safe to Resume? Breaking Execution Continuity of Agent Execution via Rollback](https://mlsanity.com/paper/2608.29381/) (2026)
- [Synthetic Computers at Scale for Long-Horizon Productivity Simulation](https://mlsanity.com/paper/2604.28181/) (2026)
- [ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction](https://mlsanity.com/paper/2605.11212/) (2026)
- [Callability Is Not Operability: Controlled Interface Interventions for LLM Agents](https://mlsanity.com/paper/2608.23628/) (2026)
- [EchoPath: Execution-Level Replayable Memory for GUI Agents](https://mlsanity.com/paper/2609.16635/) (2026)
- [Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools](https://mlsanity.com/paper/2609.39018/) (2026)
- [OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents](https://mlsanity.com/paper/2604.24348/) (2026)
- [Syll: Open-Source Personal Automation with Cross-Surface Execution](https://mlsanity.com/paper/2606.07594/) (2026)
- [World Models in Pieces: Structural Certification for General Agents](https://mlsanity.com/paper/2606.24842/) (2026)
- [Learning Latency-Aware Orchestration for Multi-Agent Systems](https://mlsanity.com/paper/2607.13359/) (2026)

## Abstract (arXiv)

Generative AI is being leveraged to solve a variety of computer-use tasks involving desktop applications. State-of-the-art systems have focused solely on improving accuracy on leading benchmarks. However, these systems are practically unusable due to extremely high end-to-end latency (e.g., tens of minutes) for tasks that typically take humans just a few minutes to complete. To understand the cause behind this and to guide future developments of computer agents, we conduct the first study on the temporal performance of computer-use agents on OSWorld, the flagship benchmark in computer-use AI. We find that large model calls for planning, reflection, and judging account for most of the overall latency, and as an agent uses more steps to complete a task, each successive step can take 3x longer than steps at the beginning of a task. We then construct OSWorld Human, a manually annotated version of the original OSWorld dataset that contains a human-determined trajectory for each task. We evaluate 16 agents on their efficiency using OSWorld Human and found that even the best agents take 2.7-4.3x more steps than necessary.