mlsanityData to 7 Oct 2026

UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback

arXiv 2406.07739 ↗First posted 11 Jun 2024cs.CL cs.HC cs.SEcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expensive human feedback or distilling a proprietary model. In this paper, we explore the use of automated feedback (compilers and multi-modal models) to guide LLMs to generate high-quality UI code. Our method starts with an existing LLM and iteratively produces improved models by self-generating a large synthetic dataset using an original model, applying automated tools to aggressively filter, score, and de-duplicate the data into a refined higher quality dataset. The original LLM is improved by finetuning on this refined dataset. We applied our approach to several open-source LLMs and compared the resulting performance to baseline models with both automated metrics and human preferences. Our evaluation shows the resulting models outperform all other downloadable baselines and approach the performance of larger proprietary models.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 44 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

25 of 44 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023StarCoder: may the source be with you!cited by 790
13 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  5. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 130
  6. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  7. 2020Attribute-conditioned Layout GAN for Automatic Graphic Designcited by 67
  8. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  9. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  10. 2022On the Transferability of Pre-trained Language Models for Low-Resource Programming Languagescited by 30
  11. 2023PLay: Parametrically Conditioned Layout Generation using Latent Diffusioncited by 24
  12. 2020Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraintscited by 11
  13. 2020Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoderscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  2. 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 12
  3. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  4. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  5. 2026VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generationcited by 7
  6. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 6
  7. 2026AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Designcited by 5
  8. 2025Advancing AI-Scientist Understanding: Multi-Agent LLMs with Interpretable Physics Reasoningcited by 4
  9. 2026WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testingcited by 4
  10. 2024Frontend Diffusion: Exploring Intent-Based User Interfaces through Abstract-to-Detailed Task Transitionscited by 3
  11. 2025Engineering Systems for Data Analysis Using Interactive Structured Inductive Programmingcited by 3
  12. 2025Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understandingcited by 3
14 more
  1. 2025DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfacescited by 3
  2. 2025Improving User Interface Generation Models from Designer Feedbackcited by 3
  3. 2026Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generationcited by 2
  4. 20261D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-Worldcited by 1
  5. 2026UXBench: Measuring the Actionability of LLM-Generated UX Critiquescited by 1
  6. 2025ControlGUI: Guiding Generative GUI Exploration through Perceptual Visual Flowcited by 0
  7. 2025When Discourse Stalls: Moving Past Five Semantic Stopsigns about Generative AI in Design Researchcited by 0
  8. 2026LikeThis! Empowering App Users to Submit UI Improvement Suggestions Instead of Complaintscited by 0
  9. 2026Efficient Personalization of Generative User Interfacescited by 0
  10. 2026Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generationcited by 0
  11. 2026LandingAgent: A Reference-Annotated Dataset and Agentic Generation Framework for Landing Pagescited by 0
  12. 2026The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agentscited by 0
  13. 2026Enabling Creative Exploration for Vibe Design Agentscited by 0
  14. 2026TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation
  2. 2024CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences
  3. 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset
  4. 2024Generating Automatic Feedback on UI Mockups with Large Language Models
  5. 2025BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
  6. 2025LOCOFY Large Design Models -- Design to code conversion solution
  7. 2025UI-UG: A Unified MLLM for UI Understanding and Generation
  8. 2026SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation