# WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch

arXiv: https://arxiv.org/abs/2505.03733 · HTML: https://mlsanity.com/paper/2505.03733/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 48 of its 65 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 48)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT](https://mlsanity.com/paper/2505.00703/) (2025)
- [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025)
- [MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine](https://mlsanity.com/paper/2407.08739/) (2024)
- [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025)
- [MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines](https://mlsanity.com/paper/2409.12959/) (2024)
- [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024)
- [SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution](https://mlsanity.com/paper/2501.05040/) (2025)

## Built on by (most cited first, 30 of 72)

- [ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation](https://mlsanity.com/paper/2507.04952/) (2025)
- [InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training](https://mlsanity.com/paper/2601.04126/) (2026)
- [WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics](https://mlsanity.com/paper/2601.02430/) (2026)
- [Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification](https://mlsanity.com/paper/2603.26648/) (2026)
- [WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning](https://mlsanity.com/paper/2509.22644/) (2025)
- [UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization](https://mlsanity.com/paper/2511.08195/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback](https://mlsanity.com/paper/2601.04203/) (2025)
- [Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing](https://mlsanity.com/paper/2506.16136/) (2025)
- [IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?](https://mlsanity.com/paper/2509.24709/) (2025)
- [WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games](https://mlsanity.com/paper/2605.17637/) (2026)
- [OVD: On-policy Verbal Distillation](https://mlsanity.com/paper/2601.21968/) (2026)
- [Safe and Scalable Web Agent Learning via Recreated Websites](https://mlsanity.com/paper/2603.10505/) (2026)
- [GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection](https://mlsanity.com/paper/2605.07442/) (2026)
- [BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception](https://mlsanity.com/paper/2510.09361/) (2025)
- [WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis](https://mlsanity.com/paper/2606.01869/) (2026)
- [MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants](https://mlsanity.com/paper/2603.09652/) (2026)
- [WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics](https://mlsanity.com/paper/2603.13391/) (2026)
- [Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Development](https://mlsanity.com/paper/2509.25297/) (2025)
- [Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation](https://mlsanity.com/paper/2607.08938/) (2026)
- [WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models](https://mlsanity.com/paper/2604.18224/) (2026)
- [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026)
- [MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation](https://mlsanity.com/paper/2604.15309/) (2026)
- [Alignment with Fill-In-the-Middle for Enhancing Code Generation](https://mlsanity.com/paper/2508.19532/) (2025)
- [InteractScience: Programmatic and Visually-Grounded Evaluation of Interactive Scientific Demonstration Code Generation](https://mlsanity.com/paper/2510.09724/) (2025)
- [WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing](https://mlsanity.com/paper/2603.25226/) (2026)
- [RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation](https://mlsanity.com/paper/2603.25804/) (2026)
- [SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering](https://mlsanity.com/paper/2605.17526/) (2026)
- [Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement](https://mlsanity.com/paper/2606.05920/) (2026)
- [Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding](https://mlsanity.com/paper/2505.05026/) (2025)

## Abstract (arXiv)

LLM-based agents have demonstrated great potential in generating and managing code within complex codebases. In this paper, we introduce WebGen-Bench, a novel benchmark designed to measure an LLM-based agent's ability to create multi-file website codebases from scratch. It contains diverse instructions for website generation, created through the combined efforts of human annotators and GPT-4o. These instructions span three major categories and thirteen minor categories, encompassing nearly all important types of web applications. To assess the quality of the generated websites, we use GPT-4o to generate test cases targeting each functionality described in the instructions, and then manually filter, adjust, and organize them to ensure accuracy, resulting in 647 test cases. Each test case specifies an operation to be performed on the website and the expected result after the operation. To automate testing and improve reproducibility, we employ a powerful web-navigation agent to execute tests on the generated websites and determine whether the observed responses align with the expected results. We evaluate three high-performance code-agent frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and open-source LLMs as engines. The best-performing combination, Bolt.diy powered by DeepSeek-R1, achieves only 27.8% accuracy on the test cases, highlighting the challenging nature of our benchmark. Additionally, we construct WebGen-Instruct, a training set consisting of 6,667 website-generation instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories generated from a subset of this training set achieves an accuracy of 38.2%, surpassing the performance of the best proprietary model.