# Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generation

arXiv: https://arxiv.org/abs/2401.07721 · HTML: https://mlsanity.com/paper/2401.07721/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 81 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis](https://mlsanity.com/paper/2008.05865/) (2020)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes](https://mlsanity.com/paper/1911.10949/) (2019)
- [Topological Planning with Transformers for Vision-and-Language Navigation](https://mlsanity.com/paper/2012.05292/) (2020)

## Built on by (most cited first, 5 of 5)

- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [MSD: A Benchmark Dataset for Floor Plan Generation of Building Complexes](https://mlsanity.com/paper/2407.10121/) (2024)
- [Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis](https://mlsanity.com/paper/2507.06689/) (2025)
- [GRE-Diff: Gaussian Room Embeddings for Structured Layout Diffusion](https://mlsanity.com/paper/2607.08086/) (2026)
- [Diffusion Transformers for Roof Graph Synthesis and Reconstruction](https://mlsanity.com/paper/2608.25652/) (2026)

## Abstract (arXiv)

We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for challenging graph-constrained architectural layout generation tasks. The proposed graph-Transformer-based generator includes a novel graph Transformer encoder that combines graph convolutions and self-attentions in a Transformer to model both local and global interactions across connected and non-connected graph nodes. Specifically, the proposed connected node attention (CNA) and non-connected node attention (NNA) aim to capture the global relations across connected nodes and non-connected nodes in the input graph, respectively. The proposed graph modeling block (GMB) aims to exploit local vertex interactions based on a house layout topology. Moreover, we propose a new node classification-based discriminator to preserve the high-level semantic and discriminative node features for different house components. To maintain the relative spatial relationships between ground truth and predicted graphs, we also propose a novel graph-based cycle-consistency loss. Finally, we propose a novel self-guided pre-training method for graph representation learning. This approach involves simultaneous masking of nodes and edges at an elevated mask ratio (i.e., 40%) and their subsequent reconstruction using an asymmetric graph-centric autoencoder architecture. This method markedly improves the model's learning proficiency and expediency. Experiments on three challenging graph-constrained architectural layout generation tasks (i.e., house layout generation, house roof generation, and building layout generation) with three public datasets demonstrate the effectiveness of the proposed method in terms of objective quantitative scores and subjective visual realism. New state-of-the-art results are established by large margins on these three tasks.