mlsanityData to 14 Oct 2026

Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training

arXiv 2610.05861 ↗First posted 5 Oct 2026cs.CV cs.AIcited by 0

On the map: World models → World model security and GUI agents also fits: Video-generation world models and simulators

also in 2 more
Abstract (arXiv)

Graphical User Interface (GUI) agents have emerged as a promising paradigm for automating complex digital workflows across diverse applications. However, training highly capable and generalizable agents fundamentally relies on massive, high-fidelity visual-action trajectories, which are notoriously difficult to acquire. While human demonstrations are unscalable, existing GUI world models rely on text descriptions or HTML rendering, discarding crucial pixel-level visual details like icons and layout styles. To address this issue, we introduce Infinite-Dreamer, a simulation-free data synthesis method powered by a pixel-level Image Editing World Model. By conceptualizing GUI transitions as image editing tasks, we leverage Vision-Language Models (VLMs) to describe action-induced UI changes as structured delta-text. We then fine-tune an image editing backbone to controllably synthesize realistic screenshot transitions. We utilize this model to generate both single-frame visual robustness data and multi-step imaginary trajectories. To validate the effectiveness of our approach, we fine-tune the Qwen3-VL baseline solely on the synthesized data to obtain Infinite-Actor, and evaluate it on AndroidWorld, MobileWorld, and AndroidControl-Curated benchmarks. Infinite-Actor consistently outperforms the Qwen3-VL baselines across scales: Infinite-Actor-8B improves AndroidWorld Pass@1 by +4.45 and nearly doubles the MobileWorld Pass@3 success rate, while Infinite-Actor-2B improves Pass@1 by +9.05. Code is available at https://github.com/swaydy-n/Infinite-Dreamer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 21 of its 128 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

21 of 128 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026MobileDreamer: Generative Sketch World Model for GUI Agent7 Jan

Other references · 20

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  3. 2023Mastering Diverse Domains through World Modelscited by 1,253
  4. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  5. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  6. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  7. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  8. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  9. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  10. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  11. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  12. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
8 more
  1. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 72
  2. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 21
  3. 2025Efficient Agent Training for Computer Usecited by 18
  4. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  5. 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
  6. 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 7
  7. 2025AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purificationcited by 6
  8. 2025GUIrilla: A Scalable Framework for Automated Desktop UI Explorationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agent
  3. 2026CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
  4. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
  6. 2026Benchmarking and Improving GUI Agents in High-Dynamic Environments
  7. 2026Code2World: A GUI World Model via Renderable Code Generation
  8. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control