mlsanityData to 14 Oct 2026

GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding

arXiv 2608.21832 ↗First posted 22 Aug 2026cs.CLcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset (kappa = 0.94 well-formedness; kappa = 0.79 target selection). Nineteen vision-language models reach at most 32% strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on 60-92% of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman ρ = +0.74), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 33 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

17 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedback25 Sep
  2. 2025GTA1: GUI Test-time Scaling Agent8 Jul
  3. 2025Visual Test-time Scaling for GUI Agent Grounding1 May

Other references · 14

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  3. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  6. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  7. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  8. 2022Visual Spatial Reasoningcited by 293
  9. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  10. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  11. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  12. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 37
2 more
  1. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  2. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
  2. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
  3. 2026MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search
  4. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
  5. 2025Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
  6. 2026When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning
  7. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
  8. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models