# Graph Generative Pre-trained Transformer

arXiv: https://arxiv.org/abs/2501.01073 · HTML: https://mlsanity.com/paper/2501.01073/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 84 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [DiGress: Discrete Denoising diffusion for graph generation](https://mlsanity.com/paper/2209.14734/) (2022)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders](https://mlsanity.com/paper/1802.03480/) (2018)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [Discrete Flow Matching](https://mlsanity.com/paper/2407.15595/) (2024)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)

## Built on by (most cited first, 17 of 17)

- [Generative Graph Pattern Machine](https://mlsanity.com/paper/2505.16130/) (2025)
- [Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guarantees](https://mlsanity.com/paper/2605.18775/) (2026)
- [Principled Latent Diffusion for Graphs via Laplacian Autoencoders](https://mlsanity.com/paper/2601.13780/) (2026)
- [GGBall: Graph Generative Model on Poincar'e Ball](https://mlsanity.com/paper/2506.07198/) (2025)
- [TinyGraphEstimator: Adapting Lightweight Language Models for Graph Structure Inference](https://mlsanity.com/paper/2510.08808/) (2025)
- [Flowette: Flow Matching with Graphette Priors for Graph Generation](https://mlsanity.com/paper/2602.23566/) (2026)
- [From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences](https://mlsanity.com/paper/2604.10332/) (2026)
- [PROVCREATOR: Synthesizing Complex Heterogenous Graphs with Node and Edge Attributes](https://mlsanity.com/paper/2507.20967/) (2025)
- [A document is worth a structured record: Principled inductive bias design for document recognition](https://mlsanity.com/paper/2507.08458/) (2025)
- [NGTM: Substructure-based Neural Graph Topic Model for Interpretable Graph Generation](https://mlsanity.com/paper/2507.13133/) (2025)
- [Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer](https://mlsanity.com/paper/2603.22854/) (2026)
- [Are Common Substructures Transferable? Riemannian Graph Foundation Model with Neural Vector Bundles](https://mlsanity.com/paper/2606.03270/) (2026)
- [Scaling Novel Graph Generation via Lightweight Structure-Guided Autoregressive Models](https://mlsanity.com/paper/2606.04287/) (2026)
- [Calibrating Generative Models to Feature Distributions with MMD Finetuning](https://mlsanity.com/paper/2606.19496/) (2026)
- [Communicability-Inspired Positional Encoding (CIPE)](https://mlsanity.com/paper/2606.25293/) (2026)
- [Training Graph Foundation Models on The Web Graph](https://mlsanity.com/paper/2609.30894/) (2026)
- [SNACK: A Sequential Notation Framework for Probabilistic Graph Generation](https://mlsanity.com/paper/2610.19781/) (2026)

## Abstract (arXiv)

Graph generation is a critical task in numerous domains, including molecular design and social network analysis, due to its ability to model complex relationships and structured data. While most modern graph generative models utilize adjacency matrix representations, this work revisits an alternative approach that represents graphs as sequences of node set and edge set. We advocate for this approach due to its efficient encoding of graphs and propose a novel representation. Based on this representation, we introduce the Graph Generative Pre-trained Transformer (G2PT), an auto-regressive model that learns graph structures via next-token prediction. To further exploit G2PT's capabilities as a general-purpose foundation model, we explore fine-tuning strategies for two downstream applications: goal-oriented generation and graph property prediction. We conduct extensive experiments across multiple datasets. Results indicate that G2PT achieves superior generative performance on both generic graph and molecule datasets. Furthermore, G2PT exhibits strong adaptability and versatility in downstream tasks from molecular design to property prediction. Code available at https://github.com/tufts-ml/G2PT,