mlsanityData to 7 Oct 2026

Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

arXiv 2406.20098 ↗First posted 28 Jun 2024cs.CV cs.AI cs.CLcited by 73

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio in a variety of understanding and generation tasks. However, current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. To address this problem, we propose textttWeb2Code, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs. For dataset construction, we leverage pretrained LLMs to enhance existing webpage-to-code datasets as well as generate a diverse pool of new webpages rendered into images. Specifically, the inputs are webpage images and instructions, while the responses are the webpage's HTML code. We further include diverse natural language QA pairs about the webpage content in the responses to enable a more comprehensive understanding of the web content. To evaluate model performance in these tasks, we develop an evaluation framework for testing MLLMs' abilities in webpage understanding and web-to-code generation. Extensive experiments show that our proposed dataset is beneficial not only to our proposed tasks but also in the general visual domain. We hope our work will contribute to the development of general MLLMs suitable for web-based content generation and task automation. Our data and code are available at https://github.com/MBZUAI-LLM/web2code.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 62 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

44 of 62 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset14 Mar

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
31 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  13. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  14. 2024InternLM2 Technical Reportcited by 391
  15. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  16. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  17. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  18. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  19. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  20. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  21. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  22. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  23. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  24. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113
  25. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  26. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  27. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  28. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  29. 2024Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewritingcited by 11
  30. 2024Assessing, Exploiting, and Mitigating Syntactic Robustness Failures in LLM-Based Code Generationcited by 6
  31. 2024CodeCloak: A Method for Evaluating and Mitigating Code Leakage by LLM Code Assistantscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 36.

73 papers.

Sort:

By area:

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 50
  6. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  7. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 41
  8. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  9. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflowcited by 25
  12. 2024MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designscited by 23
38 more
  1. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 18
  2. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 18
  3. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  4. 2025P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmarkcited by 15
  5. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 14
  6. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalizationcited by 13
  7. 2025VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Modelscited by 13
  8. 2026Generative Visual Code Mobile World Modelscited by 13
  9. 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 12
  10. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  11. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 11
  12. 2025IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?cited by 10
  13. 2025PhysicsArena: The First Multimodal Physics Reasoning Benchmark Exploring Variable, Process, and Solution Dimensionscited by 8
  14. 2026WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubricscited by 8
  15. 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 7
  16. 2026MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistantscited by 7
  17. 2026WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Modelscited by 6
  18. 2025EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generationcited by 4
  19. 2025UI-UG: A Unified MLLM for UI Understanding and Generationcited by 4
  20. 2026MM-ReCoder: Advancing Chart-to-Code Generation with Reinforcement Learning and Self-Correctioncited by 4
  21. 2025OpusAnimation: Code-Based Dynamic Chart Generationcited by 3
  22. 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
  23. 2026Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generationcited by 3
  24. 2026I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applicationscited by 3
  25. 2025Multimodal graph representation learning for website generation based on visual sketchcited by 2
  26. 2025Multilingual Multimodal Software Developer for Code Generationcited by 2
  27. 2025MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUscited by 2
  28. 2026SVRepair: Structured Visual Reasoning for Automated Program Repaircited by 2
  29. 2026Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generationcited by 2
  30. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
  31. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
  32. 2025InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generationcited by 1
  33. 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
  34. 2026InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?cited by 1
  35. 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpagescited by 1
  36. 2026Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifactscited by 1
  37. 2026UXBench: Measuring the Actionability of LLM-Generated UX Critiquescited by 1
  38. 2026UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimizationcited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
  2. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
  3. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
  4. 2024Harnessing Webpage UIs for Text-Rich Visual Understanding
  5. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code
  6. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
  7. 2025WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluation
  8. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search