# On the Reliability of Computer Use Agents

arXiv: https://arxiv.org/abs/2604.17849 · HTML: https://mlsanity.com/paper/2604.17849/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 12 of 12)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Bayesian Preference Elicitation with Language Models](https://mlsanity.com/paper/2403.05534/) (2024)
- [Active Preference Inference using Language Models and Probabilistic Reasoning](https://mlsanity.com/paper/2312.12009/) (2023)
- [WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point](https://mlsanity.com/paper/2502.08047/) (2025)
- [APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs](https://mlsanity.com/paper/2410.19656/) (2024)
- [ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environments](https://mlsanity.com/paper/2601.02399/) (2025)
- [Leveraging In-Context Learning for Language Model Agents](https://mlsanity.com/paper/2506.13109/) (2025)

## Built on by (most cited first, 10 of 10)

- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents](https://mlsanity.com/paper/2608.06663/) (2026)
- [AndroidReality: How Far Are Mobile Agents from the Real World?](https://mlsanity.com/paper/2608.07775/) (2026)
- [The Agent Use of Agent Beings: Agent Cybernetics Is the Missing Science of Foundation Agents](https://mlsanity.com/paper/2605.10754/) (2026)
- [DENSE: Distilling Agent Trajectories into Evidence-Grounded Shortcut Trees for Self-Refinement](https://mlsanity.com/paper/2609.21423/) (2026)
- [SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents](https://mlsanity.com/paper/2609.35596/) (2026)
- [Neuro-Symbolic Computer Use: Learning Reusable Policies for Reliable and Efficient Execution](https://mlsanity.com/paper/2609.36927/) (2026)
- [Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets](https://mlsanity.com/paper/2609.38914/) (2026)
- [GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution](https://mlsanity.com/paper/2610.00948/) (2026)
- [A Trust Layer for Agent Evaluation](https://mlsanity.com/paper/2610.07274/) (2026)

## Abstract (arXiv)

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.