# PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels

arXiv: https://arxiv.org/abs/2504.16419 · HTML: https://mlsanity.com/paper/2504.16419/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [AutoGLM: Autonomous Foundation Agents for GUIs](https://mlsanity.com/paper/2411.00820/) (2024)
- [OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition](https://mlsanity.com/paper/2403.19128/) (2024)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms](https://mlsanity.com/paper/2410.18967/) (2024)
- [WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics](https://mlsanity.com/paper/2301.13280/) (2023)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?](https://mlsanity.com/paper/2008.05132/) (2020)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset](https://mlsanity.com/paper/2407.08850/) (2024)
- [MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling](https://mlsanity.com/paper/2405.07090/) (2024)

## Built on by (most cited first, 3 of 3)

- [GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL](https://mlsanity.com/paper/2602.22190/) (2026)
- [SAMTok: Representing Any Mask with Two Words](https://mlsanity.com/paper/2601.16093/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) datasets are crucial for various downstream tasks. However, GUI datasets often generate annotation information through automatic labeling, which commonly results in inaccurate GUI element BBox annotations, including missing, duplicate, or meaningless BBoxes. These issues can degrade the performance of models trained on these datasets, limiting their effectiveness in real-world applications. Additionally, existing GUI datasets only provide BBox annotations visually, which restricts the development of visually related GUI downstream tasks. To address these issues, we introduce PixelWeb, a large-scale GUI dataset containing over 100,000 annotated web pages. PixelWeb is constructed using a novel automatic annotation approach that integrates visual feature extraction and Document Object Model (DOM) structure analysis through two core modules: channel derivation and layer analysis. Channel derivation ensures accurate localization of GUI elements in cases of occlusion and overlapping elements by extracting BGRA four-channel bitmap annotations. Layer analysis uses the DOM to determine the visibility and stacking order of elements, providing precise BBox annotations. Additionally, PixelWeb includes comprehensive metadata such as element images, contours, and mask annotations. Manual verification by three independent annotators confirms the high quality and accuracy of PixelWeb annotations. Experimental results on GUI element detection tasks show that PixelWeb achieves performance on the mAP95 metric that is 3-7 times better than existing datasets. We believe that PixelWeb has great potential for performance improvement in downstream tasks such as GUI generation and automated user interaction.