Advancing vision-language models in front-end development via data synthesis
On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection
also in 1 more
Abstract (arXiv)
Modern front-end (FE) development, especially when leveraging the unique features of frameworks like React and Vue, presents distinctive challenges. These include managing modular architectures, ensuring synchronization between data and visual outputs for declarative rendering, and adapting reusable components to various scenarios. Such complexities make it particularly difficult for state-of-the-art large vision-language models (VLMs) to generate accurate and functional code directly from design images. To address these challenges, we propose a reflective agentic workflow that synthesizes high-quality image-text data to capture the diverse characteristics of FE development. This workflow automates the extraction of self-containedfootnoteA self-contained code snippet is one that encapsulates all necessary logic, styling, and dependencies, ensuring it functions independently without requiring external imports or context. code snippets from real-world projects, renders the corresponding visual outputs, and generates detailed descriptions that link design elements to functional code. To further expand the scope and utility of the synthesis, we introduce three data synthesis strategies: Evolution-based synthesis, which enables scalable and diverse dataset expansion; Waterfall-Model-based synthesis, which generates logically coherent code derived from system requirements; and Additive Development synthesis, which iteratively increases the complexity of human-authored components. We build a large vision-language model, Flame, trained on the synthesized datasets and demonstrate its effectiveness in generating React code via the pass@k metric. Our results suggest that a code VLM trained to interpret images before code generation may achieve better performance.
Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 3 Mar 2025.
Builds on
38 of 47 references.
References · 38
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Visual Instruction Tuningcited by 7,792
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
26 more
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2022Competition-Level Code Generation with AlphaCodecited by 1,427
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
- 2024Qwen2.5-Coder Technical Reportcited by 1,151
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
- 2023ChatDev: Communicative Agents for Software Developmentcited by 893
- 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
- 2023StarCoder: may the source be with you!cited by 790
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
- 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
- 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
- 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
- 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
- 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
- 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
- 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
- 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
- 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
- 2024Bridging Design and Development with Automated Declarative UI Code Generationcited by 18
- 2024Stable Code Technical Reportcited by 16
Built on by
Citing papers per year: 2025: 4; 2026: 12.
16 papers.
Sort:
By area:
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
- 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
- 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
- 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
- 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generationcited by 8
- 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
- 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpagescited by 1
- 2026JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agentscited by 1
- 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verificationcited by 1
- 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generationcited by 0
4 more
- 2026Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generationcited by 0
- 2026MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modificationcited by 0
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 0
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelitycited by 0
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity7 Oct 2026
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation25 Aug 2026
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generation12 Aug 2026
- 2026MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification4 Aug 2026
- 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification30 Jul 2026
- 2026JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents26 Jul 2026
- 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation7 Jul 2026
- 2026Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation29 May 2026
- 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages29 May 2026
- 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents29 Apr 2026
- 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMs11 Mar 2026
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World20 Feb 2026
4 more
- 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization11 Nov 2025
- 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation9 Nov 2025
- 2025A Survey on Agentic Multimodal Large Language Models13 Oct 2025
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun 2025
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
- 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
- 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpagescited by 1
- 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verificationcited by 1
- 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification30 Jul 2026
- 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages29 May 2026
- 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMs11 Mar 2026
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun 2025
- 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
- 2026MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modificationcited by 0
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelitycited by 0
- 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity7 Oct 2026
- 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation25 Aug 2026
- 2026MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification4 Aug 2026
- 2025A Survey on Agentic Multimodal Large Language Models13 Oct 2025
- 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
- 2026JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agentscited by 1
- 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generationcited by 0
- 2026Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generationcited by 0
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 0
- 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generation12 Aug 2026
- 2026JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents26 Jul 2026
- 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation7 Jul 2026
- 2026Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation29 May 2026
- 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World20 Feb 2026
- 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization11 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation
- 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
- 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
- 2026Vision-Guided Iterative Refinement for Frontend Code Generation
- 2025VSA:Visual-Structural Alignment for UI-to-Code
- 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach
- 2026Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
- 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset