textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
Multimodal large language models (MLLMs) have markedly expanded the competence of graphical user-interface (GUI) systems, propelling them beyond controlled simulations into complex, real-world environments across diverse platforms. However, practical usefulness is still bounded by the reliability of visual grounding, i.e., mapping textual references to exact on-screen elements. This limitation prevents the system from accurately performing pointer-level actions such as clicking or dragging. To address it, we introduce GUI-Spotlight -- a model trained for image-grounded reasoning that dynamically invokes multiple specialized tools to iteratively narrow its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy. On the ScreenSpot-Pro benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8% accuracy, surpassing V2P-7B (50.6% with 9.6M training samples) and GTA-1-7B (50.1% with 1.56M training samples).
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Oct 2025.
Builds on
27 of 37 references.
Closest ancestors · 6
Same direction, newest first
- 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning6 Aug
- 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning26 May
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
Other references · 21
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2025Group Sequence Policy Optimizationcited by 788
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
- 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 281
- 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
9 more
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
- 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
- 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
- 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
- 2025Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agentcited by 10
- 2024Visual Grounding with Attention-Driven Constraint Balancingcited by 7
- 2025InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interactioncited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 7.
8 papers.
- 2026Continual GUI Agentscited by 7
- 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedbackcited by 5
- 2026Avenir-Web: Human-Experience-Imitating Multimodal Web Agents with Mixture of Grounding Expertscited by 4
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 2
- 2026Benchmarking and Improving GUI Agents in High-Dynamic Environmentscited by 1
- 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Groundingcited by 1
- 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 1
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 1
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding29 Jun 2026
- 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs10 May 2026
- 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding4 May 2026
- 2026Benchmarking and Improving GUI Agents in High-Dynamic Environments28 Apr 2026
- 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding23 Apr 2026
- 2026Avenir-Web: Human-Experience-Imitating Multimodal Web Agents with Mixture of Grounding Experts2 Feb 2026
- 2026Continual GUI Agents28 Jan 2026
- 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback1 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
- 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
- 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
- 2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
- 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
- 2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding
- 2026GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
- 2026Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching