Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Test-Time Scaling (TTS), which samples multiple candidate actions and ranks them via a Critic Model, has emerged as a promising paradigm for generalist GUI agents. Its efficacy thus hinges on the critic's fine-grained ranking ability. However, existing GUI critic models uniformly adopt binary classification. Our motivational analysis of these models exposes a severe entanglement: scores for valid actions and plausible-but-invalid distractors become indistinguishable. We attribute this failure to two structural defects: Affordance Collapse--the hierarchical affordance space is compressed into 0/1 labels; and Noise Sensitivity--binary objectives overfit to noisy decision boundaries. To resolve this, we introduce BBCritic (Beyond-Binary Critic), a paradigm shift grounded in the Functional Equivalence Hypothesis. Through two-stage contrastive learning, BBCritic aligns instructions and actions in a shared Affordance Space, recovering the hierarchical structure that binary supervision flattens. We also present BBBench (Beyond-Binary Bench), the first GUI critic benchmark that pairs a dense action space with a hierarchical four-level taxonomy, enabling fine-grained ranking evaluation. Experimental results show that BBCritic-3B, trained without any extra annotation, outperforms 7B-parameter SOTA binary models. It demonstrates strong zero-shot transferability across platforms and tasks, supporting our methodological view: GUI critique is fundamentally a metric-learning problem, not a classification one.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 65 references that are in the map. First posted on arXiv 14 May 2026.
Builds on
53 of 65 references.
Closest ancestors · 7
Same direction, newest first
- 2026GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models26 Jan
- 2025GTA1: GUI Test-time Scaling Agent8 Jul
- 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation5 Jun
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models22 Feb
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction5 Dec
Other references · 46
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2025Qwen3 Technical Reportcited by 8,820
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
34 more
- 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
- 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
- 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
- 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
- 2024On the Effects of Data Scale on UI Control Agentscited by 194
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
- 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
- 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
- 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
- 2024LLM Critics Help Catch LLM Bugscited by 147
- 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
- 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
- 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
- 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
- 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
- 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 35
- 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 34
- 2025HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Modelscited by 32
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
- 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
- 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agentscited by 20
- 2025ProBench: Benchmarking GUI Agents with Accurate Process Informationcited by 15
- 2025KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generationcited by 9
- 2025Self-Improving VLM Judges Without Human Annotationscited by 7
- 2025D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomaliescited by 6
- 2025OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Modelscited by 6
- 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worldscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Learn from Move: the Next Step for GUI Agents
- 2026VisCritic: Visual State Comparison as Process Reward for GUI Agents
- 2026EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
- 2026Learning Reliable GUI Agents under Imperfect Priors
- 2026MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
- 2026Benchmarking and Improving GUI Agents in High-Dynamic Environments
- 2025NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks
- 2026Latent World Action Model for GUI Agents