mlsanityData to 7 Oct 2026

Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

arXiv 2403.09029 ↗First posted 14 Mar 2024cs.HC cs.AI cs.CVcited by 126

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 5 of its 11 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

5 of 11 references.

References · 5

  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  3. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  4. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  5. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 49; 2026: 53.

126 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024What matters when building vision-language models?cited by 321
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  5. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  6. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  7. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  8. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  9. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  10. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  11. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  12. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
38 more
  1. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  2. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  4. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  5. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  6. 2025Deep Research: A Survey of Autonomous Research Agentscited by 63
  7. 2025FineVision: Open Data Is All You Needcited by 51
  8. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 50
  9. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 43
  10. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  11. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 41
  12. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  13. 2025Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedbackcited by 31
  14. 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototypingcited by 29
  15. 2026STEP3-VL-10B Technical Reportcited by 28
  16. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  17. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  18. 2024MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designscited by 23
  19. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 23
  20. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  21. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  22. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 19
  23. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 18
  24. 2024HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Modelscited by 16
  25. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Webcited by 16
  26. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  27. 2026Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verificationcited by 16
  28. 2025P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmarkcited by 15
  29. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  30. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Codecited by 14
  31. 2025FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedbackcited by 14
  32. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  33. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  34. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  35. 2026DM0: An Embodied-Native Vision-Language-Action Model towards Physical AIcited by 12
  36. 2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
  37. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
  38. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understandingcited by 10

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
  2. 2025Advancing vision-language models in front-end development via data synthesis
  3. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?
  4. 2024Enhancing Vision-Language Pre-training with Rich Supervisions
  5. 2026Generative Visual Code Mobile World Models
  6. 2026TransAnyText: Translating Arbitrary Text in E-commerce Images via Structured Visual Generation
  7. 2025WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluation
  8. 2024Dual-View Visual Contextualization for Web Navigation