mlsanityData to 14 Oct 2026

DOne: Decoupling Structure and Rendering for High-Fidelity Design-to-Code Generation

arXiv 2604.01226 ↗First posted 12 Mar 2026cs.CV cs.SEcited by 0

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

While Vision Language Models (VLMs) have shown promise in Design-to-Code generation, they suffer from a "holistic bottleneck-failing to reconcile high-level structural hierarchy with fine-grained visual details, often resulting in layout distortions or generic placeholders. To bridge this gap, we propose DOne, an end-to-end framework that decouples structure understanding from element rendering. DOne introduces (1) a learned layout segmentation module to decompose complex designs, avoiding the limitations of heuristic cropping; (2) a specialized hybrid element retriever to handle the extreme aspect ratios and densities of UI components; and (3) a schema-guided generation paradigm that bridges layout and code. To rigorously assess performance, we introduce HiFi2Code, a benchmark featuring significantly higher layout complexity than existing datasets. Extensive evaluations on the HiFi2Code demonstrate that DOne outperforms exiting methods in both high-level visual similarity (e.g., over 10% in GPT Score) and fine-grained element alignment. Human evaluations confirm a 3 times productivity gain with higher visual fidelity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 40 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

19 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs28 Jun
  2. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset14 Mar

Other references · 17

  1. 2020End-to-End Object Detection with Transformerscited by 7,414
  2. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  4. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 515
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  6. 2023Self-collaboration Code Generation via ChatGPTcited by 258
  7. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
  8. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36
  9. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  10. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  11. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agentscited by 26
  12. 2024Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approachcited by 24
5 more
  1. 2024MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designscited by 24
  2. 2025UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designscited by 22
  3. 2022Using Deep Learning to Generate Complete Log Statementscited by 17
  4. 2018Automated Reporting of GUI Design Violations for Mobile Appscited by 15
  5. 2023Exploring Distributional Shifts in Large Language Models for Code Analysiscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models
  2. 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World
  3. 2025VSA:Visual-Structural Alignment for UI-to-Code
  4. 2024VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction
  5. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation
  6. 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity
  7. 2026Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement
  8. 2023Unifying Layout Generation with a Decoupled Diffusion Model