# WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation

arXiv: https://arxiv.org/abs/2511.06251 · HTML: https://mlsanity.com/paper/2511.06251/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 14 of 14)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping](https://mlsanity.com/paper/2411.03292/) (2024)
- [WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs](https://mlsanity.com/paper/2404.06369/) (2024)
- [FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow](https://mlsanity.com/paper/2505.17399/) (2025)
- [ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents](https://mlsanity.com/paper/2507.22827/) (2025)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [Bridging Design and Development with Automated Declarative UI Code Generation](https://mlsanity.com/paper/2409.11667/) (2024)
- [MLLM-Based UI2Code Automation Guided by UI Layout Information](https://mlsanity.com/paper/2506.10376/) (2025)
- [Advancing vision-language models in front-end development via data synthesis](https://mlsanity.com/paper/2503.01619/) (2025)
- [WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development](https://mlsanity.com/paper/2410.18362/) (2024)

## Built on by (most cited first, 10 of 10)

- [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026)
- [Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation](https://mlsanity.com/paper/2605.30000/) (2026)
- [WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts](https://mlsanity.com/paper/2606.03220/) (2026)
- [Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation](https://mlsanity.com/paper/2608.24138/) (2026)
- [1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World](https://mlsanity.com/paper/2602.18548/) (2026)
- [Video2Code: Generating Interactive Webpages from UI Videos via Action-Aware Revisit](https://mlsanity.com/paper/2606.20711/) (2026)
- [Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards](https://mlsanity.com/paper/2607.04727/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings](https://mlsanity.com/paper/2606.17727/) (2026)
- [TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity](https://mlsanity.com/paper/2610.10374/) (2026)

## Abstract (arXiv)

User interface (UI) development requires translating design mockups into functional code, a process that remains repetitive and labor-intensive. While recent Vision-Language Models (VLMs) automate UI-to-Code generation, they generate only static HTML/CSS/JavaScript layouts lacking interactivity. To address this, we propose WebVIA, the first agentic framework for interactive UI-to-Code generation and validation. The framework comprises three components: 1) an exploration agent to capture multi-state UI screenshots; 2) a UI2Code model that generates executable interactive code; 3) a validation module that verifies the interactivity. Experiments demonstrate that WebVIA-Agent achieves more stable and accurate UI exploration than general-purpose agents (e.g., Gemini-2.5-Pro). In addition, our fine-tuned WebVIA-UI2Code models exhibit substantial improvements in generating executable and interactive HTML/CSS/JavaScript code, outperforming their base counterparts across both interactive and static UI2Code benchmarks. Our code and models are available at hrefhttps://zheny2751-dotcom.github.io/webvia.github.io/texttthttps://webvia.github.io.