mlsanityData to 14 Oct 2026

The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models

arXiv 2609.32679 ↗First posted 26 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: World models → World model security and GUI agents

Abstract (arXiv)

GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 15 of its 40 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

15 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug
  2. 2026How Mobile World Model Guides GUI Agents?11 May
  3. 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb
  4. 2026MobileDreamer: Generative Sketch World Model for GUI Agent7 Jan
  5. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents16 Dec

Other references · 10

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  4. 2026Kimi K3: Open Frontier Intelligencecited by 138
  5. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  6. 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigationcited by 13
  7. 2025PAL-UI: Planning with Active Look-back for Vision-Based GUI Agentscited by 11
  8. 2025UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environmentscited by 5
  9. 2026HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agentscited by 3
  10. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agentscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
  2. 2026Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
  3. 2026EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
  4. 2026Scaling GUI Agents with Visual State Transitions
  5. 2026ScratchWorld: Evaluating If World Models Compute Executable Consequences
  6. 2026Computer-Using World Model
  7. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
  8. 2026GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions