# The Art of Building Verifiers for Computer Use Agents arXiv: https://arxiv.org/abs/2604.06240 · HTML: https://mlsanity.com/paper/2604.06240/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. - Builds on 18 of its 30 references that are in the map. - First posted on arXiv 5 Apr 2026. ## Placed in - LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/) ## Builds on (most cited first, 18 of 18) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023) - [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022) - [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024) - [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024) - [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025) - [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024) - [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025) - [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025) - [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026) - [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024) - [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://mlsanity.com/paper/2504.08942/) (2025) - [Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning](https://mlsanity.com/paper/2505.14069/) (2025) - [AgentRx: Diagnosing AI Agent Failures from Execution Trajectories](https://mlsanity.com/paper/2602.02475/) (2026) - [NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild](https://mlsanity.com/paper/2410.02907/) (2024) - [A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models](https://mlsanity.com/paper/2510.08049/) (2025) - [Fara-7B: An Efficient Agentic Model for Computer Use](https://mlsanity.com/paper/2511.19663/) (2025) ## Built on by (most cited first, 13 of 13) - [AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents](https://mlsanity.com/paper/2607.02599/) (2026) - [StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure](https://mlsanity.com/paper/2607.11388/) (2026) - [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026) - [Fara-1.5: Scalable Learning Environments for Computer Use Agents](https://mlsanity.com/paper/2606.20785/) (2026) - [DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents](https://mlsanity.com/paper/2605.17439/) (2026) - [Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents](https://mlsanity.com/paper/2607.12397/) (2026) - [Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale](https://mlsanity.com/paper/2607.28074/) (2026) - [Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation](https://mlsanity.com/paper/2606.24515/) (2026) - [SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents](https://mlsanity.com/paper/2607.23263/) (2026) - [Process-Reward Tactic Evolution for Long-Horizon Bioinformatics Workflows](https://mlsanity.com/paper/2606.20839/) (2026) - [Vestrum: Improving Agent Harnesses by Adapting Their Verification, Structure and Memory](https://mlsanity.com/paper/2609.33822/) (2026) - [SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety](https://mlsanity.com/paper/2609.36201/) (2026) - [Finding Blind Spots in AppWorld and WorkArena Task Verifiers](https://mlsanity.com/paper/2610.09142/) (2026) ## Abstract (arXiv) Verifying the success of computer use agent (CUA) trajectories is a critical challenge: without reliable verification, neither evaluation nor training signal can be trusted. In this paper, we present lessons learned from building a best-in-class verifier for web tasks we call the Universal Verifier. We design the Universal Verifier around four key principles: 1) constructing rubrics with meaningful, non-overlapping criteria to reduce noise; 2) separating process and outcome rewards that yield complementary signals, capturing cases where an agent follows the right steps but gets blocked or succeeds through an unexpected path; 3) distinguishing between controllable and uncontrollable failures scored via a cascading-error-free strategy for finer-grained failure understanding; and 4) a divide-and-conquer context management scheme that attends to all screenshots in a trajectory, improving reliability on longer task horizons. We validate these findings on CUAVerifierBench, a new set of CUA trajectories with both process and outcome human labels, showing that our Universal Verifier agrees with humans as often as humans agree with each other. We report a reduction in false positive rates to near zero compared to baselines like WebVoyager (≥ 45%) and WebJudge (≥ 22%). We emphasize that these gains stem from the cumulative effect of the design choices above. We also find that an auto-research agent achieves 70% of expert quality in 5% of the time, but fails to discover all strategies required to replicate the Universal Verifier. We open-source our Universal Verifier system along with CUAVerifierBench; available at https://github.com/microsoft/fara.