# UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention

arXiv: https://arxiv.org/abs/2212.03440 · HTML: https://mlsanity.com/paper/2212.03440/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Latent Predictor Networks for Code Generation](https://mlsanity.com/paper/1603.06744/) (2016)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning](https://mlsanity.com/paper/2003.00380/) (2020)
- [Sketch2code: Generating a website from a paper mockup](https://mlsanity.com/paper/1905.13750/) (2019)
- [Detecting Visual Relationships Using Box Attention](https://mlsanity.com/paper/1807.02136/) (2018)
- [Owl Eyes: Spotting UI Display Issues via Visual Understanding](https://mlsanity.com/paper/2009.01417/) (2020)
- [Multimodal Icon Annotation For Mobile Applications](https://mlsanity.com/paper/2107.04452/) (2021)

## Abstract (arXiv)

Graphic User Interface (GUI) is facing great demand with the popularization and prosperity of mobile apps. Automatic UI code generation from UI design draft dramatically simplifies the development process. However, the nesting layer structure in the design draft affects the quality and usability of the generated code. Few existing GUI automated techniques detect and group the nested layers to improve the accessibility of generated code. In this paper, we proposed our UI Layers Group Detector as a vision-based method that automatically detects images (i.e., basic shapes and visual elements) and text layers that present the same semantic meanings. We propose two plug-in components, text fusion and box attention, that utilize text information from design drafts as a priori information for group localization. We construct a large-scale UI dataset for training and testing, and present a data augmentation approach to boost the detection performance. The experiment shows that the proposed method achieves a decent accuracy regarding layers grouping.