# Sketch2code: Generating a website from a paper mockup

arXiv: https://arxiv.org/abs/1905.13750 · HTML: https://mlsanity.com/paper/1905.13750/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2019.

## Built on by (most cited first, 20 of 20)

- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs](https://mlsanity.com/paper/2404.06369/) (2024)
- [Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping](https://mlsanity.com/paper/2410.16232/) (2024)
- [LaTCoder: Converting Webpage Design to Code with Layout-as-Thought](https://mlsanity.com/paper/2508.03560/) (2025)
- [VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation](https://mlsanity.com/paper/2602.05998/) (2026)
- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development](https://mlsanity.com/paper/2410.18362/) (2024)
- [VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?](https://mlsanity.com/paper/2608.10408/) (2026)
- [Text2App: A Framework for Creating Android Apps from Text Descriptions](https://mlsanity.com/paper/2104.08301/) (2021)
- [Automatic code generation from sketches of mobile applications in end-user development using Deep Learning](https://mlsanity.com/paper/2103.05704/) (2021)
- [EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning](https://mlsanity.com/paper/2309.09867/) (2023)
- [Multimodal graph representation learning for website generation based on visual sketch](https://mlsanity.com/paper/2504.18729/) (2025)
- [Sketch2FullStack: Generating Skeleton Code of Full Stack Website and Application from Sketch using Deep Learning and Computer Vision](https://mlsanity.com/paper/2211.14607/) (2022)
- [Learning UI-to-Code Reverse Generator Using Visual Critic Without Rendering](https://mlsanity.com/paper/2305.14637/) (2023)
- [UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention](https://mlsanity.com/paper/2212.03440/) (2022)
- [Reverse Browser: Vector-Image-to-Code Generator](https://mlsanity.com/paper/2509.05394/) (2025)
- [ProductWebGen: Benchmarking Multimodal Product Webpage Generation](https://mlsanity.com/paper/2606.01022/) (2026)
- [SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation](https://mlsanity.com/paper/2608.22390/) (2026)
- [MM-SVGEdit: A Multimodal-Driven SVG Editing for UI Design](https://mlsanity.com/paper/2609.06116/) (2026)

## Abstract (arXiv)

An early stage of developing user-facing applications is creating a wireframe to layout the interface. Once a wireframe has been created it is given to a developer to implement in code. Developing boiler plate user interface code is time consuming work but still requires an experienced developer. In this dissertation we present two approaches which automates this process, one using classical computer vision techniques, and another using a novel application of deep semantic segmentation networks. We release a dataset of websites which can be used to train and evaluate these approaches. Further, we have designed a novel evaluation framework which allows empirical evaluation by creating synthetic sketches. Our evaluation illustrates that our deep learning approach outperforms our classical computer vision approach and we conclude that deep learning is the most promising direction for future research.