# From Visual Widgets to UI Code: Efficient Tool-Grounded Generation

arXiv: https://arxiv.org/abs/2608.12611 · HTML: https://mlsanity.com/paper/2608.12611/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 28 of 28)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs](https://mlsanity.com/paper/2406.20098/) (2024)
- [Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts](https://mlsanity.com/paper/2210.03885/) (2022)
- [DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation](https://mlsanity.com/paper/2506.06251/) (2025)
- [WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs](https://mlsanity.com/paper/2404.06369/) (2024)
- [Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization](https://mlsanity.com/paper/2312.10165/) (2023)
- [Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information](https://mlsanity.com/paper/2401.08732/) (2024)
- [UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs](https://mlsanity.com/paper/2505.09904/) (2025)
- [MLLM-Based UI2Code Automation Guided by UI Layout Information](https://mlsanity.com/paper/2506.10376/) (2025)
- [Advancing vision-language models in front-end development via data synthesis](https://mlsanity.com/paper/2503.01619/) (2025)
- [UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization](https://mlsanity.com/paper/2511.08195/) (2025)
- [Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation](https://mlsanity.com/paper/2506.17307/) (2025)
- [How to Train the Teacher Model for Effective Knowledge Distillation](https://mlsanity.com/paper/2407.18041/) (2024)
- [DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models](https://mlsanity.com/paper/2506.13663/) (2025)
- [LaTCoder: Converting Webpage Design to Code with Layout-as-Thought](https://mlsanity.com/paper/2508.03560/) (2025)
- [Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs](https://mlsanity.com/paper/2512.19918/) (2025)
- [ASMIL: Attention-Stabilized Multiple Instance Learning for Whole Slide Imaging](https://mlsanity.com/paper/2603.06658/) (2026)
- [WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development](https://mlsanity.com/paper/2410.18362/) (2024)
- [UI-UG: A Unified MLLM for UI Understanding and Generation](https://mlsanity.com/paper/2509.24361/) (2025)

## Built on by (most cited first, 1 of 1)

- [ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling](https://mlsanity.com/paper/2610.15166/) (2026)

## Abstract (arXiv)

Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce WidgetGen, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (JSX). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and (1,000) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.