# Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

arXiv: https://arxiv.org/abs/2610.11942 · HTML: https://mlsanity.com/paper/2610.11942/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World model security and GUI agents, one of 11 directions in World models.
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Placed in

- World models / [World model security and GUI agents](https://mlsanity.com/wm/world-model-security-and-gui-agents/)
- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 28 of 28)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [The Value Equivalence Principle for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2011.03506/) (2020)
- [ViMo: A Generative Visual GUI World Model for App Agents](https://mlsanity.com/paper/2504.13936/) (2025)
- [RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents](https://mlsanity.com/paper/2506.00618/) (2025)
- [Code2World: A GUI World Model via Renderable Code Generation](https://mlsanity.com/paper/2602.09856/) (2026)
- [MobileDreamer: Generative Sketch World Model for GUI Agent](https://mlsanity.com/paper/2601.04035/) (2026)
- [OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows](https://mlsanity.com/paper/2510.24411/) (2025)
- [MobileWorldBench: Towards Semantic World Modeling For Mobile Agents](https://mlsanity.com/paper/2512.14014/) (2025)
- [DECEPTICON: How Dark Patterns Manipulate Web Agents](https://mlsanity.com/paper/2512.22894/) (2025)
- [Generative Visual Code Mobile World Models](https://mlsanity.com/paper/2602.01576/) (2026)
- [It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents](https://mlsanity.com/paper/2512.23128/) (2025)
- [How Mobile World Model Guides GUI Agents?](https://mlsanity.com/paper/2605.10347/) (2026)
- [Mind the Gap: Action Rebinding Attacks against Android GUI Agents](https://mlsanity.com/paper/2601.12349/) (2026)
- [AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions](https://mlsanity.com/paper/2605.25707/) (2026)
- [Scaling GUI Agents with Visual State Transitions](https://mlsanity.com/paper/2607.24112/) (2026)
- [Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces](https://mlsanity.com/paper/2606.13686/) (2026)
- [SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction](https://mlsanity.com/paper/2607.15550/) (2026)
- [AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents](https://mlsanity.com/paper/2608.05891/) (2026)
- [MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models](https://mlsanity.com/paper/2606.04627/) (2026)

## Abstract (arXiv)

A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.