WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
User interface (UI) development requires translating design mockups into functional code, a process that remains repetitive and labor-intensive. While recent Vision-Language Models (VLMs) automate UI-to-Code generation, they generate only static HTML/CSS/JavaScript layouts lacking interactivity. To address this, we propose WebVIA, the first agentic framework for interactive UI-to-Code generation and validation. The framework comprises three components: 1) an exploration agent to capture multi-state UI screenshots; 2) a UI2Code model that generates executable interactive code; 3) a validation module that verifies the interactivity. Experiments demonstrate that WebVIA-Agent achieves more stable and accurate UI exploration than general-purpose agents (e.g., Gemini-2.5-Pro). In addition, our fine-tuned WebVIA-UI2Code models exhibit substantial improvements in generating executable and interactive HTML/CSS/JavaScript code, outperforming their base counterparts across both interactive and static UI2Code benchmarks. Our code and models are available at hrefhttps://zheny2751-dotcom.github.io/webvia.github.io/texttthttps://webvia.github.io.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 22 references that are in the map. First posted on arXiv 9 Nov 2025.
Builds on
14 of 22 references.
Closest ancestors · 1
Same direction, newest first
Other references · 13
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
- 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
- 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
- 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 51
- 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 44
- 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 28
- 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
- 2024Bridging Design and Development with Automated Declarative UI Code Generationcited by 19
- 2025MLLM-Based UI2Code Automation Guided by UI Layout Informationcited by 17
- 2025Advancing vision-language models in front-end development via data synthesiscited by 16
1 more
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 6
- 2026Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generationcited by 3
- 2026WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifactscited by 2
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
- 2026Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisitcited by 1
- 2026Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboardscited by 1
- 2026Agentic Visual Generation: From Generative Models to Agentic Controlcited by 1
- 2026LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settingscited by 0
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelitycited by 0
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity7 Oct 2026
- 2026Agentic Visual Generation: From Generative Models to Agentic Control6 Sep 2026
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation25 Aug 2026
- 2026Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards6 Jul 2026
- 2026Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisit16 Jun 2026
- 2026LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings16 Jun 2026
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence14 Jun 2026
- 2026WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts2 Jun 2026
- 2026Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation28 May 2026
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World20 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping
- 2025WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning
- 2026VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
- 2026Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
- 2026WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness
- 2025Portal UX Agent -- A Plug-and-Play Engine for Rendering UIs from Natural Language Specifications
- 2026Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification
- 2025FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback