VisCritic: Visual State Comparison as Process Reward for GUI Agents
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 2 more
Abstract (arXiv)
GUI agents powered by vision-language models show strong potential for automating digital tasks, yet frequently fail in long-horizon scenarios due to the absence of step-level verification. Existing process reward models verify actions through textual reasoning alone, missing the visual nature of GUI state changes. We introduce VisCritic, a visual process reward framework that verifies agent actions by directly comparing pre-action and post-action screenshots in visual feature space. VisCritic employs a Siamese vision transformer to extract change-aware representations, coupled with an Action-Aware Critic Head that jointly evaluates action success, task progress, and error type. A critic-training data construction pipeline generates weakly supervised samples from existing trajectories without additional human labels for critic training. Experiments and offline analyses across five benchmarks demonstrate that VisCritic serves as a plug-and-play enhancement for diverse GUI agents, generally improving benchmark metrics while providing visual diagnostic cues.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 50 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
44 of 50 references.
Closest ancestors · 4
Same direction, newest first
- 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation5 Jun
- 2025Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation22 Apr
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 40
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2023GPT-4 Technical Reportcited by 18,132
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2023Visual Instruction Tuningcited by 7,792
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
28 more
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
- 2018Fully Convolutional Siamese Networks for Change Detectioncited by 305
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
- 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
- 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
- 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
- 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
- 2026Computer-Using World Modelcited by 16
- 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agentscited by 12
- 2026Adaptive Milestone Reward for GUI Agentscited by 11
- 2025GUI-PRA: Process Reward Agent for GUI Taskscited by 8
- 2025Dyna-Mind: Learning to Simulate from Experience for Better AI Agentscited by 7
- 2025CRAFT-GUI: Curriculum-Reinforced Agent For GUI Taskscited by 6
- 2026Visual Inception: Compromising Long-term Planning in Agentic Recommenders via Multimodal Memory Poisoningcited by 4
- 2026Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbationscited by 3
- 2026Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decompositioncited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
- 2026Scaling GUI Agents with Visual State Transitions
- 2026GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models
- 2026SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation
- 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux
- 2026EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
- 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
- 2026Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training