mlsanityData to 14 Oct 2026

Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

arXiv 2607.04334 ↗First posted 5 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a document object model or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing benchmarks score task success, element grounding, or attack resistance and do not ask whether that belief is drawn from the pixels. We formalize visual state reliance, the attribution of a state belief to pixels, structure, or priors, and measure it with paired single-channel interventions over 735 probes spanning real web, mobile, and desktop interfaces, of which 225 are zero-edit divergences mined from live production websites, all scored by deterministic forced choice with no model judge. Our central metric is the Perception-Fusion Gap (PFG), the fraction of probes a model perceives correctly yet resolves toward structure under conflict; a stricter variant that re-verifies perception on a tight crop of the target region leaves the gap intact. Across models from four vendors, textual state beliefs defer to structure while image-only accuracy stays near ceiling, and on unedited stale snapshots from live pages the same models follow the outdated structure on up to 0.88 of probes. A white-box ablation traces the textual effect to a single copied structural value, and gradient attribution shows the visual evidence is processed yet overridden. In live multi-step environments, one mis-sourced belief at the first step compounds into task failure with a self-recovery rate of at most 0.03. Comparing four mitigations on identical probes, prompt-level cues fail at the action level, certificate checks buy safety with refusals, and a training-free consistency gate is alone in reducing both hijack and task error. Visual state reliance thus gives a measurable diagnostic of whether agent state beliefs are visually grounded.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 29 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

24 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
  2. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  4. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 17

  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  3. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  4. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  5. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  6. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  7. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 111
  8. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  9. 2025WebInject: Prompt Injection Attack to Web Agentscited by 51
  10. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 45
  11. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understandingcited by 43
  12. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
5 more
  1. 2026MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memorycited by 14
  2. 2025EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attackscited by 9
  3. 2026Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agentscited by 7
  4. 2026Hallucination as Exploit: Evidence-Carrying Multimodal Agentscited by 2
  5. 2026Grounding Driving VLA via Inverse Kinematicscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents
  2. 2026Agent-Computer Observation Interfaces Enable Dynamic Computer Use
  3. 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
  4. 2026See2Think: Do Multimodal Models Really Use Intermediate Visual States?
  5. 2026From Web to Pixels: Bringing Agentic Search into Visual Perception
  6. 2025Towards Understanding Graphical Perception in Large Multimodal Models
  7. 2025XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level
  8. 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents