mlsanityData to 14 Oct 2026

Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

arXiv 2610.11942 ↗First posted 8 Oct 2026cs.CVcited by 0

On the map: World models → World model security and GUI agents

also in 1 more
Abstract (arXiv)

A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

28 of 36 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug
  2. 2026SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction17 Jul
  3. 2026How Mobile World Model Guides GUI Agents?11 May
  4. 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb
  5. 2026Generative Visual Code Mobile World Models2 Feb
  6. 2026MobileDreamer: Generative Sketch World Model for GUI Agent7 Jan
  7. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents16 Dec
  8. 2025ViMo: A Generative Visual GUI World Model for App Agents15 Apr

Other references · 20

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  4. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  5. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  6. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  7. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  8. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  9. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  10. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  11. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 94
  12. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 35
8 more
  1. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 19
  2. 2025DECEPTICON: How Dark Patterns Manipulate Web Agentscited by 15
  3. 2025It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agentscited by 10
  4. 2026Mind the Gap: Action Rebinding Attacks against Android GUI Agentscited by 6
  5. 2026AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptionscited by 4
  6. 2026Scaling GUI Agents with Visual State Transitionscited by 4
  7. 2026Benchmarking Web Agent Safety under E-commerce Deceptive Interfacescited by 3
  8. 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
  2. 2026Computer-Using World Model
  3. 2026Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
  4. 2026Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
  5. 2026Temporal UI State Inconsistency in Desktop GUI Agents: Formalizing and Defending Against TOCTOU Attacks on Computer-Use Agents
  6. 2026EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
  7. 2025Towards Trustworthy GUI Agents: A Survey
  8. 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation