mlsanityData to 14 Oct 2026

WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development

arXiv 2410.18362 ↗First posted 24 Oct 2024cs.SE cs.CL cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

21 of 42 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
9 more
  1. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  3. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
  4. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 112
  5. 2021CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Treescited by 32
  6. 2019Sketch2code: Generating a website from a paper mockupcited by 20
  7. 2022HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representationscited by 9
  8. 2024CodeArt: Better Code Models by Attention Regularization When Symbols Are Lackingcited by 7
  9. 2024DocEdit-v2: Document Structure Editing Via Multimodal LLM Groundingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmarkcited by 17
  2. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generationcited by 10
  3. 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
  4. 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 1
  5. 2026SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluationcited by 0