# ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search

arXiv: https://arxiv.org/abs/2505.15259 · HTML: https://mlsanity.com/paper/2505.15259/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 29 of 29)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning](https://mlsanity.com/paper/2503.21620/) (2025)
- [GUI Agents with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2411.04890/) (2024)
- [Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs](https://mlsanity.com/paper/2402.14740/) (2024)
- [ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification](https://mlsanity.com/paper/2502.14565/) (2025)
- [KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding](https://mlsanity.com/paper/2503.12797/) (2025)
- [On the Robustness of GUI Grounding Models Against Image Attacks](https://mlsanity.com/paper/2504.04716/) (2025)
- [Breaking the Data Barrier -- Building GUI Agents Through Task Generalization](https://mlsanity.com/paper/2504.10127/) (2025)
- [Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents](https://mlsanity.com/paper/2504.17934/) (2025)

## Built on by (most cited first, 13 of 13)

- [DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning](https://mlsanity.com/paper/2507.00008/) (2025)
- [InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization](https://mlsanity.com/paper/2508.05731/) (2025)
- [IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding](https://mlsanity.com/paper/2508.09456/) (2025)
- [UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding](https://mlsanity.com/paper/2604.14113/) (2026)
- [SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control](https://mlsanity.com/paper/2508.20018/) (2025)
- [What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs](https://mlsanity.com/paper/2605.12549/) (2026)
- [MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements](https://mlsanity.com/paper/2511.13087/) (2025)
- [How Auxiliary Reasoning Unleashes GUI Grounding in VLMs](https://mlsanity.com/paper/2509.11548/) (2025)
- [One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding](https://mlsanity.com/paper/2606.30084/) (2026)
- [PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control](https://mlsanity.com/paper/2605.15963/) (2026)
- [TVWorld: Foundations for Remote-Control TV Agents](https://mlsanity.com/paper/2601.13142/) (2026)
- [Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding](https://mlsanity.com/paper/2609.15307/) (2026)
- [MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search](https://mlsanity.com/paper/2610.17792/) (2026)

## Abstract (arXiv)

Recent advances in Multimodal Large Language Models (MLLMs) have enabled autonomous agents to interact with computers via Graphical User Interfaces (GUIs), where accurately localizing the coordinates of interface elements (e.g., buttons) is often required for fine-grained actions. However, this remains significantly challenging, leading prior works to rely on large-scale web datasets to improve the grounding accuracy. In this work, we propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective framework for web grounding that enables MLLMs to learn data efficiently through self-generated reasoning and spatial-aware criticism. More specifically, ReGUIDE learns to (i) self-generate a language reasoning process for the localization via online reinforcement learning, and (ii) criticize the prediction using spatial priors that enforce equivariance under input transformations. At inference time, ReGUIDE further boosts performance through a test-time scaling strategy, which combines spatial search with coordinate aggregation. Our experiments demonstrate that ReGUIDE significantly advances web grounding performance across multiple benchmarks, outperforming baselines with substantially fewer training data points (e.g., only 0.2% samples compared to the best open-sourced baselines).