mlsanityData to 14 Oct 2026

Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping

arXiv 2411.03292 ↗First posted 5 Nov 2024cs.SE cs.AI cs.HCcited by 51

Not on the map. Why?

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failureaware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs' performance on the Interaction-toCode task. The Interaction2Code benchmark and code are available in https://github. com/WebPAI/Interaction2Code.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 20 of its 67 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

20 of 67 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  6. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  7. 2024Pixtral 12Bcited by 179
  8. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  9. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
  10. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
  11. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 74
  12. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 44
8 more
  1. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36
  2. 2024Exploring Perceptual Limitation of Multimodal Large Language Modelscited by 25
  3. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
  4. 2019Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Networkcited by 23
  5. 2024Bridging Design and Development with Automated Declarative UI Code Generationcited by 19
  6. 2018Automated Reporting of GUI Design Violations for Mobile Appscited by 15
  7. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Modelscited by 15
  8. 2017Classifying and Qualifying GUI Defectscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 32.

51 papers.

Sort:

By area:

  1. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  2. 2024MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designscited by 24
  3. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 19
  4. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 15
  5. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  6. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  7. 2025FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedbackcited by 14
  8. 2025IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?cited by 12
  9. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 12
  10. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generationcited by 10
  11. 2025LaTCoder: Converting Webpage Design to Code with Layout-as-Thoughtcited by 9
  12. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 9
38 more
  1. 2026WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesiscited by 9
  2. 2026WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubricscited by 8
  3. 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgradescited by 7
  4. 2025Generative Interfaces for Language Modelscited by 6
  5. 2025Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safetycited by 6
  6. 2026WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Modelscited by 6
  7. 2026Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Constructioncited by 6
  8. 2025EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generationcited by 5
  9. 2026MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generationcited by 5
  10. 20253D Software Synthesis Driven by Constraint-Expressive Intermediate Representationcited by 4
  11. 2025InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generationcited by 4
  12. 2026WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testingcited by 4
  13. 2025WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learningcited by 3
  14. 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Designcited by 3
  15. 2026Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generationcited by 3
  16. 2026I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applicationscited by 3
  17. 2026FullStack-Agent: Enhancing Agentic Full-Stack Web Coding via Development-Oriented Testing and Repository Back-Translationcited by 2
  18. 2026Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debuggingcited by 1
  19. 2026VulTriage: Triple-Path Context Augmentation for LLM-Based Vulnerability Detectioncited by 1
  20. 2026Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboardscited by 1
  21. 2026LiveEvalBench: Toward Open-World Evaluation for Web Generationcited by 1
  22. 2026EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistantscited by 1
  23. 2026RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agentscited by 1
  24. 2026PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slidescited by 1
  25. 2025PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agentscited by 0
  26. 2026From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generationcited by 0
  27. 2026DCVD: Dual-Channel Cross-Modal Fusion for Joint Vulnerability Detection and Localizationcited by 0
  28. 2026ProductWebGen: Benchmarking Multimodal Product Webpage Generationcited by 0
  29. 2026LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settingscited by 0
  30. 2026Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generationcited by 0
  31. 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generationcited by 0
  32. 2026CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understandingcited by 0
  33. 2026Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generationcited by 0
  34. 2026Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Developmentcited by 0
  35. 2026Editable Visual Designcited by 0
  36. 2026Enabling Creative Exploration for Vibe Design Agentscited by 0
  37. 2026ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Taskscited by 0
  38. 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agentscited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
  2. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
  3. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation
  4. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
  5. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation
  6. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code
  7. 2026InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
  8. 2026WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts