# Improved GUI Grounding via Iterative Narrowing

arXiv: https://arxiv.org/abs/2411.13591 · HTML: https://mlsanity.com/paper/2411.13591/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 13 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 11 of 11)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Iterative Visual Reasoning Beyond Convolutions](https://mlsanity.com/paper/1803.11189/) (2018)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)

## Built on by (most cited first, 15 of 15)

- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning](https://mlsanity.com/paper/2507.00008/) (2025)
- [OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution](https://mlsanity.com/paper/2601.20380/) (2026)
- [Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding](https://mlsanity.com/paper/2512.05941/) (2025)
- [UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding](https://mlsanity.com/paper/2604.14113/) (2026)
- [Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback](https://mlsanity.com/paper/2512.01979/) (2025)
- [FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agents](https://mlsanity.com/paper/2508.09241/) (2025)
- [Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding](https://mlsanity.com/paper/2604.21268/) (2026)
- [FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting](https://mlsanity.com/paper/2604.27974/) (2026)
- [Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding](https://mlsanity.com/paper/2509.24133/) (2025)
- [What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs](https://mlsanity.com/paper/2605.12549/) (2026)
- [How Auxiliary Reasoning Unleashes GUI Grounding in VLMs](https://mlsanity.com/paper/2509.11548/) (2025)
- [When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents](https://mlsanity.com/paper/2609.16732/) (2026)
- [ScreenHaystack: Finding Blind Zones in GUI Grounding](https://mlsanity.com/paper/2609.32036/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) grounding plays a crucial role in enhancing the capabilities of Vision-Language Model (VLM) agents. While general VLMs, such as GPT-4V, demonstrate strong performance across various tasks, their proficiency in GUI grounding remains suboptimal. Recent studies have focused on fine-tuning these models specifically for zero-shot GUI grounding, yielding significant improvements over baseline performance. We introduce a visual prompting framework that employs an iterative narrowing mechanism to further improve the performance of both general and fine-tuned models in GUI grounding. For evaluation, we tested our method on a comprehensive benchmark comprising various UI platforms and provided the code to reproduce our results.