UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization
Not on the map. Why?
Abstract (arXiv)
UI-to-code aims to translate UI screenshots into executable front-end code. Despite progress with vision-language models (VLMs), most existing methods formulate UI-to-code as a single-pass generation, which mismatches real-world UI development that is inherently iterative and feedback-driven. We reformulate UI-to-code as an interactive visual optimization problem, where code generation is embedded in a closed-loop process of execution, visual inspection, and iterative refinement driven by rendered visual feedback. To address the non-differentiability of visual objectives and the noise of absolute visual evaluators, we propose Relative Visual Policy Optimization (RVPO), a preference-based reinforcement learning method that optimizes relative visual rankings among rendered candidates under execution feedback. We instantiate this paradigm in UI2Code^N, an open-source 9B model trained via continual pre-training, supervised fine-tuning, and reinforcement learning. Experiments demonstrate state-of-the-art performance on UI drafting, UI polishing, and UI editing benchmarks, even outperforming larger models, with performance consistently improving through iterative visual optimization. Our code and models are available at https://github.com/zai-org/UI2Code_N.
Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 11 Nov 2025.
Builds on
19 of 22 references.
References · 19
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
- 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
- 2025Kimi-VL Technical Reportcited by 388
- 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
- 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
- 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
- 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 98
7 more
- 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 74
- 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
- 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 44
- 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 26
- 2024Bridging Design and Development with Automated Declarative UI Code Generationcited by 19
- 2025Advancing vision-language models in front-end development via data synthesiscited by 16
- 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
Built on by
Citing papers per year: 2026: 15.
15 papers.
Sort:
By area:
- 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 34
- 2026Code2World: A GUI World Model via Renderable Code Generationcited by 29
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 6
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
- 2026Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifactscited by 1
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 1
- 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 1
- 2026Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoningcited by 0
- 2026Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Developmentcited by 0
- 2026Editable Visual Designcited by 0
- 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agentscited by 0
- 2026Reinforcement Learning from Intermediate Renders for Image-to-Code Generationcited by 0
- 2026AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards13 Oct 2026
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity7 Oct 2026
- 2026TeleGen: Improving LLM-Based Web Application Generation via Runtime Telemetry4 Oct 2026
- 2026Reinforcement Learning from Intermediate Renders for Image-to-Code Generation28 Sep 2026
- 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents15 Sep 2026
- 2026Agentic Visual Generation: From Generative Models to Agentic Control6 Sep 2026
- 2026Editable Visual Design3 Sep 2026
- 2026Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development2 Sep 2026
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation25 Aug 2026
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generation12 Aug 2026
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence14 Jun 2026
- 2026Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts9 Jun 2026
- 2026Code2World: A GUI World Model via Renderable Code Generationcited by 29
- 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb 2026
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 6
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 1
- 2026Editable Visual Designcited by 0
- 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agentscited by 0
- 2026TeleGen: Improving LLM-Based Web Application Generation via Runtime Telemetrycited by 0
- 2026AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewardscited by 0
- 2026AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards13 Oct 2026
- 2026TeleGen: Improving LLM-Based Web Application Generation via Runtime Telemetry4 Oct 2026
- 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents15 Sep 2026
- 2026Editable Visual Design3 Sep 2026
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generation12 Aug 2026
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence14 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation
- 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning
- 2026UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
- 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection
- 2026VISTA: View-Consistent Self-Verified Training for GUI Grounding
- 2026OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward