LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
Not on the map. Why?
Abstract (arXiv)
Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks, yet their performance degrades sharply on small targets, densely packed controls and out-of-distribution interfaces. We attribute this gap to a paradigmatic limitation shared by existing approaches: none of them treats a produced coordinate as a hypothesis to be reflected upon and revised under new visual evidence. This manifests as three coupled issues: 1) Lack of post-hoc reflection. The prediction is frozen at the moment of emission, leaving no internal mechanism to challenge or refine it. 2) Visual evidence decoupled from the prediction. The auxiliary visual evidence is gathered to support the upcoming coordinate rather than to scrutinise the one already committed to. 3) Refinement over views, not over predictions. The iterative zoom-in refines the inspected region instead of inheriting a previous coordinate as a spatial prior to be corrected. In this paper, we propose LookAgain, a closed-loop GUI grounder driven by post-prediction visual reflection. LookAgain reformulates grounding as a multi-turn predict-look-again-refine process with two primitives: "locate" posts a coordinate hypothesis, renders a marker on the image and appends a local patch of the predicted region. It anchors the next reasoning step to the previous prediction as a spatial prior; "confirm" accepts or reject the hypothesis and terminates the procedure. We train the LookAgain grounder with SFT on constructed reflective trajectories as a cold start, followed by GRPO with terminal grounding correctness as the sole reward. Extensive experiments show that LookAgain consistently improves performance on both refusal-aware and general GUI grounding benchmarks, achieving state-of-the-art results. Comprehensive ablations further verify the effectiveness of the proposed framework.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
34 of 42 references.
References · 34
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
- 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
- 2025GTA1: GUI Test-time Scaling Agentcited by 106
- 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 99
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
22 more
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 77
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
- 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
- 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
- 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 56
- 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
- 2025Visual Test-time Scaling for GUI Agent Groundingcited by 30
- 2025Grounding Computer Use Agents on Human Demonstrationscited by 22
- 2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 21
- 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
- 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
- 2025Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimizationcited by 13
- 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
- 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
- 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 12
- 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
- 2025V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peakingcited by 6
- 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 6
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 5
- 2026VISTA: View-Consistent Self-Verified Training for GUI Groundingcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
- 2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
- 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
- 2026GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
- 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
- 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
- 2026GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction