# WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

arXiv: https://arxiv.org/abs/2609.15387 · HTML: https://mlsanity.com/paper/2609.15387/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch](https://mlsanity.com/paper/2505.03733/) (2025)
- [ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation](https://mlsanity.com/paper/2507.04952/) (2025)
- [Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions](https://mlsanity.com/paper/2310.15780/) (2023)
- [Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks](https://mlsanity.com/paper/2505.07473/) (2025)
- [FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow](https://mlsanity.com/paper/2505.17399/) (2025)
- [FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation](https://mlsanity.com/paper/2506.13832/) (2025)
- [Automatic Web Testing using Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2103.06018/) (2021)
- [WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics](https://mlsanity.com/paper/2601.02430/) (2026)
- [AWorld: Orchestrating the Training Recipe for Agentic AI](https://mlsanity.com/paper/2508.20404/) (2025)
- [FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback](https://mlsanity.com/paper/2601.04203/) (2025)
- [WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games](https://mlsanity.com/paper/2605.17637/) (2026)
- [WebSight: A Vision-First Architecture for Robust Web Agents](https://mlsanity.com/paper/2508.16987/) (2025)
- [MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants](https://mlsanity.com/paper/2603.09652/) (2026)
- [ResTest: Targeted Coverage of Residual Not-Covered Code Using Large Language Models for Web GUI Testing](https://mlsanity.com/paper/2506.00520/) (2025)
- [WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models](https://mlsanity.com/paper/2604.18224/) (2026)
- [Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation](https://mlsanity.com/paper/2605.30000/) (2026)
- [I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applications](https://mlsanity.com/paper/2606.00750/) (2026)
- [Fragment-Based Test Generation For Web Apps](https://mlsanity.com/paper/2110.14043/) (2021)
- [Multimodal graph representation learning for website generation based on visual sketch](https://mlsanity.com/paper/2504.18729/) (2025)
- [Deep Reinforcement Learning for Automated Web GUI Testing](https://mlsanity.com/paper/2504.19237/) (2025)
- [WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts](https://mlsanity.com/paper/2606.03220/) (2026)
- [Understanding Automated Web GUI Testing: An Empirical Study Across Exploration Strategies and State Abstractions](https://mlsanity.com/paper/2606.16650/) (2026)

## Abstract (arXiv)

Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.