# Step-level Optimization for Efficient Computer-use Agents arXiv: https://arxiv.org/abs/2604.27151 ยท HTML: https://mlsanity.com/paper/2604.27151/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. - Builds on 20 of its 29 references that are in the map. - First posted on arXiv 29 Apr 2026. ## Placed in - LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/) ## Builds on (most cited first, 20 of 20) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023) - [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024) - [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025) - [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024) - [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024) - [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024) - [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025) - [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024) - [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024) - [MasRouter: Learning to Route LLMs for Multi-Agent Systems](https://mlsanity.com/paper/2502.11133/) (2025) - [OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents](https://mlsanity.com/paper/2506.16042/) (2025) - [Fara-7B: An Efficient Agentic Model for Computer Use](https://mlsanity.com/paper/2511.19663/) (2025) - [xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning](https://mlsanity.com/paper/2510.08439/) (2025) - [Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents](https://mlsanity.com/paper/2509.26539/) (2025) - [SCUBA: Salesforce Computer Use Benchmark](https://mlsanity.com/paper/2509.26506/) (2025) - [Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions](https://mlsanity.com/paper/2604.07277/) (2026) - [ANCHOR: Branch-Point Data Generation for GUI Agents](https://mlsanity.com/paper/2602.07153/) (2026) ## Built on by (most cited first, 7 of 7) - [The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World](https://mlsanity.com/paper/2608.08239/) (2026) - [GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments](https://mlsanity.com/paper/2609.00048/) (2026) - [Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments](https://mlsanity.com/paper/2608.24099/) (2026) - [Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs](https://mlsanity.com/paper/2607.28573/) (2026) - [Do Not Restart: Residual Completion for Stateful Agent Handoffs](https://mlsanity.com/paper/2609.13800/) (2026) - [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026) - [GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution](https://mlsanity.com/paper/2610.00948/) (2026) ## Abstract (arXiv) Computer-use agents provide a promising path toward general software automation because they can interact directly with arbitrary graphical user interfaces instead of relying on brittle, application-specific integrations. Despite recent advances in benchmark performance, strong computer-use agents remain expensive and slow in practice, since most systems invoke large multimodal models at nearly every interaction step. We argue that this uniform allocation of compute is fundamentally inefficient for long-horizon GUI tasks. Such trajectories are highly heterogeneous: many steps are routine and can be handled reliably by smaller, cheaper policies, while errors tend to concentrate at a relatively small number of high-risk moments. Across computer-use benchmarks, these failures repeatedly take two forms: progress stalls, where the agent loops, repeats ineffective actions, or fails to make meaningful progress, and silent semantic drift, where the agent continues taking locally plausible actions after already deviating from the user's true goal. To address this inefficiency, we propose an event-driven, step-level cascade for computer-use agents that runs a small policy by default and escalates to a stronger model only when lightweight learned monitors detect elevated risk. Our framework combines two complementary signals: a Stuck Monitor that detects degraded progress from recent reasoning-action history and triggers recovery, and a Milestone Monitor that identifies semantically meaningful checkpoints where sparse verification is most informative for catching drift. This design turns always-on frontier-model inference into adaptive, on-demand compute allocation over the course of an evolving interaction. The framework is modular and deployment-oriented: it can be layered on top of existing computer-use agents without changing the underlying agent architecture or retraining the large model.