mlsanityData to 7 Oct 2026

pix2code: Generating Code from a Graphical User Interface Screenshot

arXiv 1705.07962 ↗First posted 22 May 2017cs.LG cs.AI cs.CLcited by 113

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 6; 2024: 14; 2025: 29; 2026: 47.

113 papers.

Sort:

By area:

  1. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  2. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  3. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  4. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  5. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  6. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  7. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  8. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 50
  9. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 43
  10. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 41
  11. 2024Graphic Design with Large Multimodal Modelcited by 38
  12. 2021VINS: Visual Search for Mobile User Interface Designcited by 37
38 more
  1. 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototypingcited by 29
  2. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  3. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  4. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 23
  5. 2021GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networkscited by 17
  6. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  7. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Webcited by 16
  8. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  9. 2025VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representationcited by 15
  10. 2026WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metricscited by 15
  11. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Codecited by 14
  12. 2021Understanding Mobile GUI: from Pixel-Words to Screen-Sentencescited by 13
  13. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  14. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  15. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  16. 2026Multimodal OCR: Parse Anything from Documentscited by 12
  17. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
  18. 2025IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?cited by 10
  19. 2022Generative Colorization of Structured Mobile Web Pagescited by 8
  20. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generationcited by 8
  21. 2025Computer-Use Agents as Judges for Generative User Interfacecited by 8
  22. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 8
  23. 2021Variable Name Recovery in Decompiled Binary Code using Constrained Masked Language Modelingcited by 7
  24. 2025Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Modelscited by 7
  25. 2026VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generationcited by 7
  26. 2026MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistantscited by 7
  27. 2026Generative UI: LLMs are Effective UI Generatorscited by 7
  28. 2026WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Modelscited by 6
  29. 2026VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editingcited by 6
  30. 2026WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesiscited by 6
  31. 2021Magic Layouts: Structural Prior for Component Detection in User Interface Designscited by 5
  32. 2022Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Imagescited by 5
  33. 2025Generative Interfaces for Language Modelscited by 5
  34. 2025UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environmentscited by 5
  35. 2025Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Developmentcited by 5
  36. 2023An Empirical Investigation into the Use of Image Captioning for Automated Software Documentationcited by 4
  37. 2025EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generationcited by 4
  38. 2025UI-UG: A Unified MLLM for UI Understanding and Generationcited by 4

50 most cited of 113 All 113 (JSON)