Data to 14 Oct 2026

GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

arXiv 2506.03143 ↗First posted 3 Jun 2025cs.CL cs.AI cs.CVcited by 95

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

One of the principal challenges in building VLM-powered GUI agents is visual grounding, i.e., localizing the appropriate screen region for action execution based on both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment, inability to handle ambiguous supervision targets, and a mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. In this paper, we propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the model to propose one or more action regions in a single forward pass. In line with this, we further design a grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to previous state-of-the-art models, highlighting that GUI-Actor can endow the underlying VLM with effective grounding capabilities without compromising its general-purpose strengths.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 61 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

53 of 61 references.

Closest ancestors · 21

Same direction, newest first

9 more

Other references · 32

2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2025Qwen2.5-VL Technical Reportcited by 5,587
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319

20 more

2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
2024OmniParser for Pure Vision Based GUI Agentcited by 198
2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
2019An Empirical Study of Spatial Attention Mechanisms in Deep Networkscited by 58
2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
2025UFO2: The Desktop AgentOScited by 41
2024AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistantcited by 27
2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
2024Large Action Models: From Inception to Implementationcited by 18
2024Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuningcited by 14
2019ROI Pooled Correlation Filters for Visual Trackingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 34; 2026: 61.

95 papers.

Sort:

By area:

2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 113
2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 63
2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
2025FineVision: Open Data Is All You Needcited by 53
2026UI-Venus-1.5 Technical Reportcited by 41
2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 35
2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 27
2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 22
2025Grounding Computer Use Agents on Human Demonstrationscited by 22

38 more

2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistencycited by 21
2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 19
2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Webcited by 19
2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
2025MVP: Multiple View Prediction Improves GUI Groundingcited by 12
2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 11
2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
2025GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agentscited by 10
2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agentscited by 10
2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 10
2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agentscited by 10
2025GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Groundingcited by 8
2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Groundingcited by 8
2026OpAgent: Operator Agent for Web Navigationcited by 8
2026PatchCue: Enhancing Vision-Language Model Reasoning with Patch-Based Visual Cuescited by 8
2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 8
2025Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Networkcited by 7
2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
2026Continual GUI Agentscited by 7
2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 7
2025OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?cited by 6
2025CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learningcited by 6
2025UFO3: Weaving the Digital Agent Galaxycited by 6
2026SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibrationcited by 6
2026OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agentscited by 6
2026Fara-1.5: Scalable Learning Environments for Computer Use Agentscited by 6
2025Using GUI Agent for Electronic Design Automationcited by 5
2026GUITester: Enabling GUI Agents for Exploratory Defect Discoverycited by 5
2026Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elementscited by 5
2026Executable Agentic Memory for GUI Agentcited by 5
2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Groundingcited by 4
2025Improving GUI Grounding with Explicit Position-to-Coordinate Mappingcited by 4

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way