mlsanityData to 7 Oct 2026

Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering

arXiv 2403.03163 ↗First posted 5 Mar 2024cs.CL cs.CV cs.CYcited by 146

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development in which multimodal large language models (MLLMs) directly convert visual designs into code implementations. In this work, we construct Design2Code - the first real-world benchmark for this task. Specifically, we manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations to validate the performance ranking. To rigorously benchmark MLLMs, we test various multimodal prompting methods on frontier models such as GPT-4o, GPT-4V, Gemini, and Claude. Our fine-grained break-down metrics indicate that models mostly lag in recalling visual elements from the input webpages and generating correct layout designs.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 47 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

20 of 47 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
8 more
  1. 2023StarCoder: may the source be with you!cited by 790
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  5. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  6. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113
  7. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  8. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 41; 2026: 102.

146 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  4. 2025Paper2Poster: Towards Multimodal Poster Automation from Scientific Paperscited by 56
  5. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
  6. 2026STEP3-VL-10B Technical Reportcited by 28
  7. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
  8. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  9. 2024MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designscited by 23
  10. 2025StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputscited by 23
  11. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 23
  12. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
38 more
  1. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 19
  2. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  3. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 18
  4. 2025FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluationcited by 17
  5. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 17
  6. 2026Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verificationcited by 16
  7. 2025P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmarkcited by 15
  8. 2025FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedbackcited by 14
  9. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  10. 2026Generative Visual Code Mobile World Modelscited by 13
  11. 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 12
  12. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  13. 2026Multimodal OCR: Parse Anything from Documentscited by 12
  14. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
  15. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 11
  16. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
  17. 2025IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?cited by 10
  18. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 9
  19. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generationcited by 8
  20. 2025Computer-Use Agents as Judges for Generative User Interfacecited by 8
  21. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 8
  22. 2025ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generationcited by 7
  23. 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 7
  24. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  25. 2026BizGenEval: A Systematic Benchmark for Commercial Visual Content Generationcited by 7
  26. 2026KAT-Coder-V2 Technical Reportcited by 7
  27. 2026WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Modelscited by 6
  28. 2026VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editingcited by 6
  29. 2026WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesiscited by 6
  30. 2025Generative Interfaces for Language Modelscited by 5
  31. 2025BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Executioncited by 5
  32. 2026Any2Poster: Any-Source Poster Generation Across Modalities and Domainscited by 5
  33. 2026AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Designcited by 5
  34. 2025Programming with Pixels: Can Computer-Use Agents do Software Engineering?cited by 4
  35. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Modelscited by 4
  36. 2025EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generationcited by 4
  37. 2026WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testingcited by 4
  38. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Modelcited by 3

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation
  2. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping
  3. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation
  4. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
  5. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code
  6. 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity
  7. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
  8. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs