mlsanityData to 7 Oct 2026

Advancing vision-language models in front-end development via data synthesis

arXiv 2503.01619 ↗First posted 3 Mar 2025cs.CV cs.AI cs.CLcited by 16

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

also in 1 more
Abstract (arXiv)

Modern front-end (FE) development, especially when leveraging the unique features of frameworks like React and Vue, presents distinctive challenges. These include managing modular architectures, ensuring synchronization between data and visual outputs for declarative rendering, and adapting reusable components to various scenarios. Such complexities make it particularly difficult for state-of-the-art large vision-language models (VLMs) to generate accurate and functional code directly from design images. To address these challenges, we propose a reflective agentic workflow that synthesizes high-quality image-text data to capture the diverse characteristics of FE development. This workflow automates the extraction of self-containedfootnoteA self-contained code snippet is one that encapsulates all necessary logic, styling, and dependencies, ensuring it functions independently without requiring external imports or context. code snippets from real-world projects, renders the corresponding visual outputs, and generates detailed descriptions that link design elements to functional code. To further expand the scope and utility of the synthesis, we introduce three data synthesis strategies: Evolution-based synthesis, which enables scalable and diverse dataset expansion; Waterfall-Model-based synthesis, which generates logically coherent code derived from system requirements; and Additive Development synthesis, which iteratively increases the complexity of human-authored components. We build a large vision-language model, Flame, trained on the synthesized datasets and demonstrate its effectiveness in generating React code via the pass@k metric. Our results suggest that a code VLM trained to interpret images before code generation may achieve better performance.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

38 of 47 references.

References · 38

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
26 more
  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  9. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  12. 2023StarCoder: may the source be with you!cited by 790
  13. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  14. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  16. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  17. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  18. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  19. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  20. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  21. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  22. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  23. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  24. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  25. 2024Bridging Design and Development with Automated Declarative UI Code Generationcited by 18
  26. 2024Stable Code Technical Reportcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
  3. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  4. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  5. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generationcited by 8
  6. 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
  7. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
  8. 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
  9. 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpagescited by 1
  10. 2026JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agentscited by 1
  11. 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verificationcited by 1
  12. 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generationcited by 0
4 more
  1. 2026Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generationcited by 0
  2. 2026MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modificationcited by 0
  3. 2026From Visual Widgets to UI Code: Efficient Tool-Grounded Generationcited by 0
  4. 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation
  2. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
  3. 2025FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
  4. 2026Vision-Guided Iterative Refinement for Frontend Code Generation
  5. 2025VSA:Visual-Structural Alignment for UI-to-Code
  6. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach
  7. 2026Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
  8. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset