# Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

arXiv: https://arxiv.org/abs/2410.16232 · HTML: https://mlsanity.com/paper/2410.16232/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 83 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 21 of 21)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [At Which Training Stage Does Code Data Help LLMs Reasoning?](https://mlsanity.com/paper/2309.16298/) (2023)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning](https://mlsanity.com/paper/2405.20535/) (2024)
- [Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network](https://mlsanity.com/paper/1910.08930/) (2019)
- [Sketch2code: Generating a website from a paper mockup](https://mlsanity.com/paper/1905.13750/) (2019)
- [Frontend Diffusion: Exploring Intent-Based User Interfaces through Abstract-to-Detailed Task Transitions](https://mlsanity.com/paper/2408.00778/) (2024)
- [Automatic code generation from sketches of mobile applications in end-user development using Deep Learning](https://mlsanity.com/paper/2103.05704/) (2021)

## Built on by (most cited first, 29 of 29)

- [BaxBench: Can LLMs Generate Correct and Secure Backends?](https://mlsanity.com/paper/2502.11844/) (2025)
- [RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback](https://mlsanity.com/paper/2510.06186/) (2025)
- [VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning](https://mlsanity.com/paper/2511.00391/) (2025)
- [WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics](https://mlsanity.com/paper/2601.02430/) (2026)
- [P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark](https://mlsanity.com/paper/2505.17104/) (2025)
- [WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation](https://mlsanity.com/paper/2508.16763/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback](https://mlsanity.com/paper/2601.04203/) (2025)
- [WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games](https://mlsanity.com/paper/2605.17637/) (2026)
- [MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants](https://mlsanity.com/paper/2603.09652/) (2026)
- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows](https://mlsanity.com/paper/2504.01931/) (2025)
- [AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use](https://mlsanity.com/paper/2607.20536/) (2026)
- [WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning](https://mlsanity.com/paper/2510.04097/) (2025)
- [Vision-Guided Iterative Refinement for Frontend Code Generation](https://mlsanity.com/paper/2604.05839/) (2026)
- [Paper2Web: Let's Make Your Paper Alive!](https://mlsanity.com/paper/2510.15842/) (2025)
- [1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World](https://mlsanity.com/paper/2602.18548/) (2026)
- [Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Study](https://mlsanity.com/paper/2604.07558/) (2026)
- [Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages](https://mlsanity.com/paper/2606.00154/) (2026)
- [PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides](https://mlsanity.com/paper/2609.29718/) (2026)
- [Exploring Student Choice and the Use of Multimodal Generative AI in Programming Learning](https://mlsanity.com/paper/2510.05417/) (2025)
- [Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation](https://mlsanity.com/paper/2510.27632/) (2025)
- [CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval](https://mlsanity.com/paper/2604.15663/) (2026)
- [LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings](https://mlsanity.com/paper/2606.17727/) (2026)
- [UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation](https://mlsanity.com/paper/2607.06306/) (2026)
- [MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction](https://mlsanity.com/paper/2607.19910/) (2026)
- [Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development](https://mlsanity.com/paper/2609.02088/) (2026)
- [Beyond Static Charts: Can Language and Vision Language Models Generate Interactive Data Visualization Interfaces?](https://mlsanity.com/paper/2609.26208/) (2026)
- [AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfaces](https://mlsanity.com/paper/2609.34854/) (2026)

## Abstract (arXiv)

Sketches are a natural and accessible medium for UI designers to conceptualize early-stage ideas. However, existing research on UI/UX automation often requires high-fidelity inputs like Figma designs or detailed screenshots, limiting accessibility and impeding efficient design iteration. To bridge this gap, we introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. Beyond end-to-end benchmarking, Sketch2Code supports interactive agent evaluation that mimics real-world design workflows, where a VLM-based agent iteratively refines its generations by communicating with a simulated user, either passively receiving feedback instructions or proactively asking clarification questions. We comprehensively analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs; even the most capable models struggle to accurately interpret sketches and formulate effective questions that lead to steady improvement. Nevertheless, a user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception, highlighting the need to develop more effective paradigms for multi-turn conversational agents.