# MLLM-Based UI2Code Automation Guided by UI Layout Information

arXiv: https://arxiv.org/abs/2506.10376 · HTML: https://mlsanity.com/paper/2506.10376/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2025.

## Built on by (most cited first, 17 of 17)

- [ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents](https://mlsanity.com/paper/2507.22827/) (2025)
- [VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning](https://mlsanity.com/paper/2511.00391/) (2025)
- [WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation](https://mlsanity.com/paper/2511.06251/) (2025)
- [Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs](https://mlsanity.com/paper/2512.19918/) (2025)
- [Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Development](https://mlsanity.com/paper/2509.25297/) (2025)
- [EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation](https://mlsanity.com/paper/2509.12159/) (2025)
- [DesignAsCode: Bridging Structural Editability and Visual Fidelity in Graphic Design Generation](https://mlsanity.com/paper/2602.17690/) (2026)
- [3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation](https://mlsanity.com/paper/2507.18625/) (2025)
- [TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework](https://mlsanity.com/paper/2510.17163/) (2025)
- [1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World](https://mlsanity.com/paper/2602.18548/) (2026)
- [Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging](https://mlsanity.com/paper/2604.19750/) (2026)
- [From Visual Widgets to UI Code: Efficient Tool-Grounded Generation](https://mlsanity.com/paper/2608.12611/) (2026)
- [SPRITE: From Static Mockups to Engine-Ready Game UI](https://mlsanity.com/paper/2604.18591/) (2026)
- [ProductWebGen: Benchmarking Multimodal Product Webpage Generation](https://mlsanity.com/paper/2606.01022/) (2026)
- [LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings](https://mlsanity.com/paper/2606.17727/) (2026)
- [Decomposer: Learning to Decompile Symbolic Music to Programs](https://mlsanity.com/paper/2607.01849/) (2026)
- [Editable Visual Design](https://mlsanity.com/paper/2609.04034/) (2026)

## Abstract (arXiv)

Converting user interfaces into code (UI2Code) is a crucial step in website development, which is time-consuming and labor-intensive. The automation of UI2Code is essential to streamline this task, beneficial for improving the development efficiency. There exist deep learning-based methods for the task; however, they heavily rely on a large amount of labeled training data and struggle with generalizing to real-world, unseen web page designs. The advent of Multimodal Large Language Models (MLLMs) presents potential for alleviating the issue, but they are difficult to comprehend the complex layouts in UIs and generate the accurate code with layout preserved. To address these issues, we propose LayoutCoder, a novel MLLM-based framework generating UI code from real-world webpage images, which includes three key modules: (1) Element Relation Construction, which aims at capturing UI layout by identifying and grouping components with similar structures; (2) UI Layout Parsing, which aims at generating UI layout trees for guiding the subsequent code generation process; and (3) Layout-Guided Code Fusion, which aims at producing the accurate code with layout preserved. For evaluation, we build a new benchmark dataset which involves 350 real-world websites named Snap2Code, divided into seen and unseen parts for mitigating the data leakage issue, besides the popular dataset Design2Code. Extensive evaluation shows the superior performance of LayoutCoder over the state-of-the-art approaches. Compared with the best-performing baseline, LayoutCoder improves 10.14% in the BLEU score and 3.95% in the CLIP score on average across all datasets.