GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments
Not on the map. Why?
Abstract (arXiv)
An image generation model used as a graphical user interface (GUI) environment must produce not only a plausible screen but also the correct response to an action. Existing visual-generation benchmarks do not directly assess this combination of functional correctness and temporal consistency in GUIs. We introduce GUI-GenBench, a benchmark of 700 curated samples across five task suites: single-step transitions, multi-step planning, fictional-app generation, real-app trajectories, and coordinate-based grounding. We also introduce GUI-Score, a VLM-based metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Evaluating 12 image generation models on Chinese and English subsets, we find that strong single-step performance does not consistently extend to coherent multi-step trajectories or accurate grounding. The highest aggregate GUI-Scores are 69.62 on the Chinese subset and 63.16 on the English subset. Qualitative analysis further identifies failures in text rendering, icon interpretation, and coordinate-conditioned transitions. Together, these results distinguish visual plausibility from functional reliability and identify limitations that must be addressed before generated GUIs can serve as dependable interaction environments.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
46 of 61 references.
References · 46
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2024GPT-4o System Cardcited by 4,467
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2025Qwen-Image Technical Reportcited by 973
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
34 more
- 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
- 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
- 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
- 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 233
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
- 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
- 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
- 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
- 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
- 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
- 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
- 2025LongCat-Image Technical Reportcited by 92
- 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
- 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
- 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
- 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
- 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
- 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 41
- 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
- 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
- 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
- 2025UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesiscited by 20
- 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 18
- 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
- 2021GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networkscited by 17
- 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 12
- 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editingcited by 12
- 2023Boosting GUI Prototyping with Diffusion Modelscited by 10
- 2022GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Networkcited by 6
- 2024Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generationcited by 4
- 2025GUIDE: LLM-Driven GUI Generation Decomposition for Automated Prototypingcited by 4
- 2025GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learningcited by 4
- 2023GUILGET: GUI Layout GEneration with Transformercited by 2
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubricscited by 8
- 2026VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine Systemcited by 7
- 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 1
- 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide userscited by 0
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments30 Aug 2026
- 2026VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System29 Jul 2026
- 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide users23 Apr 2026
- 2026WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics11 Mar 2026
- 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation9 Mar 2026