mlsanityData to 7 Oct 2026

GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents

arXiv 2601.09770 ↗First posted 14 Jan 2026cs.AIcited by 9

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 44 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

35 of 44 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Training25 Jun
  2. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
  3. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  4. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  6. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding23 Sep
  7. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  8. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 27

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  10. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  11. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  12. 2023Language Models can Solve Computer Taskscited by 371
15 more
  1. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  2. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  3. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  4. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  5. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  6. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  7. 2025Visual Planning: Let's Think Only with Imagescited by 85
  8. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  9. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  10. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  11. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  12. 2024AI Flow at the Network Edgecited by 43
  13. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  14. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
  15. 2025ScaleTrack: Scaling and back-tracking Automated GUI Agentscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
  2. 2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents
  3. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  4. 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
  5. 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
  6. 2025Learning Active Perception via Self-Evolving Preference Optimization for GUI Grounding
  7. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  8. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding