# VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation

arXiv: https://arxiv.org/abs/2602.05998 · HTML: https://mlsanity.com/paper/2602.05998/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [LongCoder: A Long-Range Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2306.14893/) (2023)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation](https://mlsanity.com/paper/2501.06598/) (2025)
- [Rendering-Aware Reinforcement Learning for Vector Graphics Generation](https://mlsanity.com/paper/2505.20793/) (2025)
- [UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback](https://mlsanity.com/paper/2406.07739/) (2024)
- [V-Thinker: Interactive Thinking with Images](https://mlsanity.com/paper/2511.04460/) (2025)
- [UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs](https://mlsanity.com/paper/2505.09904/) (2025)
- [VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning](https://mlsanity.com/paper/2511.00391/) (2025)
- [Sketch2code: Generating a website from a paper mockup](https://mlsanity.com/paper/1905.13750/) (2019)
- [Bridging Design and Development with Automated Declarative UI Code Generation](https://mlsanity.com/paper/2409.11667/) (2024)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)

## Built on by (most cited first, 9 of 9)

- [Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation](https://mlsanity.com/paper/2608.24138/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [The reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agents](https://mlsanity.com/paper/2608.28795/) (2026)
- [Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development](https://mlsanity.com/paper/2609.02088/) (2026)
- [ViCo: Visual-oriented Coding with Self-Reflection for Chart Replication](https://mlsanity.com/paper/2609.16014/) (2026)
- [Reinforcement Learning from Intermediate Renders for Image-to-Code Generation](https://mlsanity.com/paper/2609.34587/) (2026)
- [TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity](https://mlsanity.com/paper/2610.10374/) (2026)
- [ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling](https://mlsanity.com/paper/2610.15166/) (2026)
- [AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards](https://mlsanity.com/paper/2610.18860/) (2026)

## Abstract (arXiv)

Screenshot-to-code generation aims to translate user interface screenshots into executable frontend code that faithfully reproduces the target layout and style. Existing multimodal large language models perform this mapping directly from screenshots but are trained without observing the visual outcomes of their generated code. In contrast, human developers iteratively render their implementation, compare it with the design, and learn how visual differences relate to code changes. Inspired by this process, we propose VisRefiner, a training framework that enables models to learn from visual differences between rendered predictions and reference designs. We construct difference-aligned supervision that associates visual discrepancies with corresponding code edits, allowing the model to understand how appearance variations arise from implementation changes. Building on this, we introduce a reinforcement learning stage for self-refinement, where the model improves its generated code by observing both the rendered output and the target design, identifying their visual differences, and updating the code accordingly. Experiments show that VisRefiner substantially improves single-step generation quality and layout fidelity, while also endowing models with strong self-refinement ability. These results demonstrate the effectiveness of learning from visual differences for advancing screenshot-to-code generation.