mlsanityData to 7 Oct 2026

Magic Layouts: Structural Prior for Component Detection in User Interface Designs

arXiv 2106.07615 ↗First posted 14 Jun 2021cs.CV cs.HC cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screenshots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 17 of its 40 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

17 of 40 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2017Mask R-CNNcited by 7,259
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2015Fast R-CNNcited by 2,930
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  9. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  10. 2017CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noisecited by 181
  11. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  12. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113
5 more
  1. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  2. 2017Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networkcited by 92
  3. 2016The More You Know: Using Knowledge Graphs for Image Classificationcited by 63
  4. 2018Iterative Visual Reasoning Beyond Convolutionscited by 38
  5. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024From Grounding to Planning: Benchmarking Bottlenecks in Web Agentscited by 12
  2. 2025InfoDet: A Dataset for Infographic Element Detectioncited by 4
  3. 2024Self-supervised Photographic Image Layout Representation Learningcited by 3
  4. 2026CoGen: Creation of Reusable UI Components in Figma via Textual Commandscited by 2
  5. 2026Detect Anything in Graphic Design: Element-Level Rewards for Autoregressive Detectioncited by 1