mlsanityData to 7 Oct 2026

Scene Graph Conditioning in Latent Diffusion

arXiv 2310.10338 ↗First posted 16 Oct 2023cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models excel in image generation but lack detailed semantic control using text prompts. Additional techniques have been developed to address this limitation. However, conditioning diffusion models solely on text-based descriptions is challenging due to ambiguity and lack of structure. In contrast, scene graphs offer a more precise representation of image content, making them superior for fine-grained control and accurate synthesis in image generation models. The amount of image and scene-graph data is sparse, which makes fine-tuning large diffusion models challenging. We propose multiple approaches to tackle this problem using ControlNet and Gated Self-Attention. We were able to show that using out proposed methods it is possible to generate images from scene graphs with much higher quality, outperforming previous methods. Our source code is publicly available on https://github.com/FrankFundel/SGCond

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 38 of its 49 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

38 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis28 Apr
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
20 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2016Generative Adversarial Text to Image Synthesiscited by 735
  9. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  10. 2018Image Generation from Scene Graphscited by 324
  11. 2022Semantic Image Synthesis via Diffusion Modelscited by 131
  12. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  13. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 103
  14. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 82
  15. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  16. 2019PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graphcited by 44
  17. 2020Image Colorization: A Survey and Datasetcited by 23
  18. 2017Multi-Stage Variational Auto-Encoders for Coarse-to-Fine Image Generationcited by 18
  19. 2023Transformer-based Image Generation from Scene Graphscited by 8
  20. 2021Unsupervised Image Generation with Infinite Generative Adversarial Networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Dependency-Aware Discrete Diffusion for Scene Graph Generation
  2. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
  3. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  4. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  5. 2026Towards Controllable Image Generation through Representation-Conditioned Diffusion Models
  6. 2024Diffusion Models Need Visual Priors for Image Generation
  7. 2025Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis
  8. 2023Modulating Pretrained Diffusion Models for Multimodal Image Synthesis