GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 44 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
35 of 44 references.
Closest ancestors · 8
Same direction, newest first
- 2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Training25 Jun
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding23 Sep
- 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 27
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2023Language Models can Solve Computer Taskscited by 371
15 more
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
- 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
- 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
- 2025Visual Planning: Let's Think Only with Imagescited by 85
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
- 2025A Survey on (M)LLM-Based GUI Agentscited by 61
- 2024AI Flow at the Network Edgecited by 43
- 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
- 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
- 2025ScaleTrack: Scaling and back-tracking Automated GUI Agentscited by 16
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 9
- 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 5
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026Interactive Reward Agent: GUI Task Evaluation via Environment-State Verificationcited by 2
- 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 1
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 1
- 2026CI4A: Semantic Component Interfaces for Agents Empowering Web Automationcited by 0
- 2026PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editorscited by 0
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agentscited by 0
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents15 Sep 2026
- 2026Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification28 Jul 2026
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding29 Jun 2026
- 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs10 May 2026
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding1 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding15 Apr 2026
- 2026PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors14 Apr 2026
- 2026CI4A: Semantic Component Interfaces for Agents Empowering Web Automation21 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
- 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
- 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
- 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
- 2025Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding
- 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding