# pix2code: Generating Code from a Graphical User Interface Screenshot

arXiv: https://arxiv.org/abs/1705.07962 · HTML: https://mlsanity.com/paper/1705.07962/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2017.

## Built on by (most cited first, 30 of 114)

- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents](https://mlsanity.com/paper/2401.00812/) (2024)
- [Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs](https://mlsanity.com/paper/2406.20098/) (2024)
- [Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping](https://mlsanity.com/paper/2411.03292/) (2024)
- [DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation](https://mlsanity.com/paper/2506.06251/) (2025)
- [WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs](https://mlsanity.com/paper/2404.06369/) (2024)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping](https://mlsanity.com/paper/2410.16232/) (2024)
- [FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow](https://mlsanity.com/paper/2505.17399/) (2025)
- [ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents](https://mlsanity.com/paper/2507.22827/) (2025)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics](https://mlsanity.com/paper/2601.02430/) (2026)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web](https://mlsanity.com/paper/2409.18980/) (2024)
- [VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models](https://mlsanity.com/paper/2508.09945/) (2025)
- [WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation](https://mlsanity.com/paper/2508.16763/) (2025)
- [VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation](https://mlsanity.com/paper/2511.02778/) (2025)
- [WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code](https://mlsanity.com/paper/2506.07818/) (2025)
- [WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning](https://mlsanity.com/paper/2509.22644/) (2025)
- [UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization](https://mlsanity.com/paper/2511.08195/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Multimodal OCR: Parse Anything from Documents](https://mlsanity.com/paper/2603.13032/) (2026)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?](https://mlsanity.com/paper/2509.24709/) (2025)

## Abstract (arXiv)

Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).