mlsanityData to 14 Oct 2026

Improved GUI Grounding via Iterative Narrowing

arXiv 2411.13591 ↗First posted 18 Nov 2024cs.CV cs.AI cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Graphical User Interface (GUI) grounding plays a crucial role in enhancing the capabilities of Vision-Language Model (VLM) agents. While general VLMs, such as GPT-4V, demonstrate strong performance across various tasks, their proficiency in GUI grounding remains suboptimal. Recent studies have focused on fine-tuning these models specifically for zero-shot GUI grounding, yielding significant improvements over baseline performance. We introduce a visual prompting framework that employs an iterative narrowing mechanism to further improve the performance of both general and fine-tuned models in GUI grounding. For evaluation, we tested our method on a comprehensive benchmark comprising various UI platforms and provided the code to reproduce our results.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 13 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

11 of 13 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  2. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  3. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 8

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  6. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  7. 2018Iterative Visual Reasoning Beyond Convolutionscited by 38
  8. 2021Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  3. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 31
  4. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  5. 2025Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Groundingcited by 8
  6. 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 8
  7. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedbackcited by 7
  8. 2025FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agentscited by 5
  9. 2026Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Groundingcited by 5
  10. 2026FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Settingcited by 5
  11. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Groundingcited by 4
  12. 2026What Happens Before Decoding? Prefill Determines GUI Grounding in VLMscited by 4
3 more
  1. 2025How Auxiliary Reasoning Unleashes GUI Grounding in VLMscited by 2
  2. 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agentscited by 0
  3. 2026ScreenHaystack: Finding Blind Zones in GUI Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
  2. 2025textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding
  3. 2026GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
  4. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback
  5. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  6. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
  7. 2026Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
  8. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection