mlsanityData to 7 Oct 2026

ScreenHaystack: Finding Blind Zones in GUI Grounding

arXiv 2609.32036 ↗First posted 25 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

21 of 24 references.

References · 21

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  6. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  7. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  8. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  9. 2024GUI Agents: A Surveycited by 101
  10. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  11. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  12. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
9 more
  1. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
  2. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  3. 2025Visual Test-time Scaling for GUI Agent Groundingcited by 27
  4. 2025Grounding Computer Use Agents on Human Demonstrationscited by 16
  5. 2024Improved GUI Grounding via Iterative Narrowingcited by 15
  6. 2025On the Robustness of GUI Grounding Models Against Image Attackscited by 14
  7. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 10
  8. 2026Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agentscited by 5
  9. 2026GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Modelscited by 2

Built on by

Not cited yet.