# Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces

arXiv: https://arxiv.org/abs/2404.13521 · HTML: https://mlsanity.com/paper/2404.13521/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 81 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Generalization and Representational Limits of Graph Neural Networks](https://mlsanity.com/paper/2002.06157/) (2020)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Mapping Natural Language Commands to Web Elements](https://mlsanity.com/paper/1808.09132/) (2018)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints](https://mlsanity.com/paper/2001.05424/) (2020)
- [GRIDS: Interactive Layout Design with Integer Programming](https://mlsanity.com/paper/2001.02921/) (2020)
- [Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders](https://mlsanity.com/paper/2001.05308/) (2020)
- [ORC Layout: Adaptive GUI Layout with OR-Constraints](https://mlsanity.com/paper/1912.07827/) (2019)
- [ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints](https://mlsanity.com/paper/2002.09925/) (2020)
- [ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints](https://mlsanity.com/paper/2202.11523/) (2022)

## Built on by (most cited first, 6 of 6)

- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)
- [Multimodal graph representation learning for website generation based on visual sketch](https://mlsanity.com/paper/2504.18729/) (2025)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)
- [UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrieval](https://mlsanity.com/paper/2511.19380/) (2025)
- [A Resource-Rational Principle for Modeling Visual Attention Control](https://mlsanity.com/paper/2603.02056/) (2026)

## Abstract (arXiv)

Present-day graphical user interfaces (GUIs) exhibit diverse arrangements of text, graphics, and interactive elements such as buttons and menus, but representations of GUIs have not kept up. They do not encapsulate both semantic and visuo-spatial relationships among elements. To seize machine learning's potential for GUIs more efficiently, Graph4GUI exploits graph neural networks to capture individual elements' properties and their semantic-visuo-spatial constraints in a layout. The learned representation demonstrated its effectiveness in multiple tasks, especially generating designs in a challenging GUI autocompletion task, which involved predicting the positions of remaining unplaced elements in a partially completed GUI. The new model's suggestions showed alignment and visual appeal superior to the baseline method and received higher subjective ratings for preference. Furthermore, we demonstrate the practical benefits and efficiency advantages designers perceive when utilizing our model as an autocompletion plug-in.