# "Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

arXiv: https://arxiv.org/abs/2511.20067 · HTML: https://mlsanity.com/paper/2511.20067/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 8 of 8)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Autonomous Evaluation and Refinement of Digital Agents](https://mlsanity.com/paper/2404.06474/) (2024)
- [SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience](https://mlsanity.com/paper/2508.04700/) (2025)
- [AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World](https://mlsanity.com/paper/2503.24278/) (2025)
- [Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation](https://mlsanity.com/paper/2508.05508/) (2025)

## Built on by (most cited first, 3 of 3)

- [StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure](https://mlsanity.com/paper/2607.11388/) (2026)
- [CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents](https://mlsanity.com/paper/2603.10577/) (2026)
- [Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation](https://mlsanity.com/paper/2606.24515/) (2026)

## Abstract (arXiv)

Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been completed. We present an autonomous evaluation and feedback framework that uses vision-language models to assess task completion directly from screenshots and task descriptions. Our dataset covers 42 built-in macOS applications and 1,260 human-labeled tasks across a wide range of scenarios. Our framework achieves up to 73 percent accuracy in task success detection and yields an average relative improvement of 27 percent in overall task success when evaluator feedback is applied. These results show that vision-based evaluation can serve as an effective feedback mechanism that improves the reliability and self-correction of autonomous computer-use agents.