# InstructG2I: Synthesizing Images from Multimodal Attributed Graphs

arXiv: https://arxiv.org/abs/2410.07157 · HTML: https://mlsanity.com/paper/2410.07157/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 46 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications](https://mlsanity.com/paper/1709.07604/) (2017)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [Fine-Grained Spoiler Detection from Large-Scale Review Corpora](https://mlsanity.com/paper/1905.13416/) (2019)

## Built on by (most cited first, 14 of 14)

- [GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs](https://mlsanity.com/paper/2502.11925/) (2025)
- [Graph4MM: Weaving Multimodal Learning with Structural Information](https://mlsanity.com/paper/2510.16990/) (2025)
- [Graph World Model](https://mlsanity.com/paper/2507.10539/) (2025)
- [Cross-Contrastive Clustering for Multimodal Attributed Graphs with Dual Graph Filtering](https://mlsanity.com/paper/2511.20030/) (2025)
- [OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph](https://mlsanity.com/paper/2602.05576/) (2026)
- [MLaGA: Multimodal Large Language and Graph Assistant](https://mlsanity.com/paper/2506.02568/) (2025)
- [Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach](https://mlsanity.com/paper/2602.04116/) (2026)
- [TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution](https://mlsanity.com/paper/2603.27723/) (2026)
- [RoleMAG: Learning Neighbor Roles in Multimodal Graphs](https://mlsanity.com/paper/2604.12271/) (2026)
- [CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregation](https://mlsanity.com/paper/2605.11468/) (2026)
- [LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learning](https://mlsanity.com/paper/2608.24795/) (2026)
- [Towards Effective Federated Multimodal Graph Learning via Navigating Multifaceted Heterogeneity](https://mlsanity.com/paper/2608.00623/) (2026)
- [Towards Multimodal Graph Large Language Model](https://mlsanity.com/paper/2506.09738/) (2025)
- [LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learning](https://mlsanity.com/paper/2601.21453/) (2026)

## Abstract (arXiv)

In this paper, we approach an overlooked yet critical task Graph2Image: generating images from multimodal attributed graphs (MMAGs). This task poses significant challenges due to the explosion in graph size, dependencies among graph entities, and the need for controllability in graph conditions. To address these challenges, we propose a graph context-conditioned diffusion model called InstructG2I. InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling by combining personalized page rank and re-ranking based on vision-language features. Then, a Graph-QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process of diffusion. Finally, we propose graph classifier-free guidance, enabling controllable generation by varying the strength of graph guidance and multiple connected edges to a node. Extensive experiments conducted on three datasets from different domains demonstrate the effectiveness and controllability of our approach. The code is available at https://github.com/PeterGriffinJin/InstructG2I.