# A History-Aware Visually Grounded Critic for Computer Use Agents

arXiv: https://arxiv.org/abs/2606.11078 · HTML: https://mlsanity.com/paper/2606.11078/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 32)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [OpenCUA: Open Foundations for Computer-Use Agents](https://mlsanity.com/paper/2508.09123/) (2025)
- [Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation](https://mlsanity.com/paper/2410.13232/) (2024)
- [AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents](https://mlsanity.com/paper/2410.24024/) (2024)
- [Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents](https://mlsanity.com/paper/2602.16855/) (2026)
- [SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience](https://mlsanity.com/paper/2508.04700/) (2025)
- [ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data](https://mlsanity.com/paper/2509.15221/) (2025)
- [Fara-7B: An Efficient Agentic Model for Computer Use](https://mlsanity.com/paper/2511.19663/) (2025)
- [Code2World: A GUI World Model via Renderable Code Generation](https://mlsanity.com/paper/2602.09856/) (2026)
- [Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation](https://mlsanity.com/paper/2506.04614/) (2025)
- [GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks](https://mlsanity.com/paper/2509.23738/) (2025)
- [GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior](https://mlsanity.com/paper/2506.08012/) (2025)
- [OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent](https://mlsanity.com/paper/2601.07779/) (2026)
- [The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement](https://mlsanity.com/paper/2503.16024/) (2025)
- [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://mlsanity.com/paper/2507.08800/) (2025)
- [Language Models Can Learn from Verbal Feedback Without Scalar Rewards](https://mlsanity.com/paper/2509.22638/) (2025)
- [CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent](https://mlsanity.com/paper/2510.18596/) (2025)
- [Policy Improvement using Language Feedback Models](https://mlsanity.com/paper/2402.07876/) (2024)
- [Retrieval-augmented GUI Agents with Generative Guidelines](https://mlsanity.com/paper/2509.24183/) (2025)
- [R-WoM: Retrieval-augmented World Model For Computer-use Agents](https://mlsanity.com/paper/2510.11892/) (2025)
- [PhyCritic: Multimodal Critic Models for Physical AI](https://mlsanity.com/paper/2602.11124/) (2026)
- [When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents](https://mlsanity.com/paper/2602.08995/) (2026)
- [RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback](https://mlsanity.com/paper/2507.15024/) (2025)
- [WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents](https://mlsanity.com/paper/2601.21872/) (2026)

## Built on by (most cited first, 1 of 1)

- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)

## Abstract (arXiv)

Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy's completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.