mlsanityData to 7 Oct 2026

Compositional Visual Generation with Composable Diffusion Models

arXiv 2206.01714 ↗First posted 3 Jun 2022cs.CV cs.AI cs.LGcited by 619

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Large text-guided diffusion models, such as DALLE-2, are able to generate stunning photorealistic images given natural language descriptions. While such models are highly flexible, they struggle to understand the composition of certain concepts, such as confusing the attributes of different objects or relations between objects. In this paper, we propose an alternative structured approach for compositional generation using diffusion models. An image is generated by composing a set of diffusion models, with each of them modeling a certain component of the image. To do this, we interpret diffusion models as energy-based models in which the data distributions defined by the energy functions may be explicitly combined. The proposed method can generate scenes at test time that are substantially more complex than those seen in training, composing sentence descriptions, object relations, human facial attributes, and even generalizing to new combinations that are rarely seen in the real world. We further illustrate how our approach may be used to compose pre-trained text-guided diffusion models and generate photorealistic images containing all the details described in the input descriptions, including the binding of certain object attributes that have been shown difficult for DALLE-2. These results point to the effectiveness of the proposed method in promoting structured generalization for visual generation. Project page: https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/

Key facts. Cited by 619 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 40 of its 60 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

40 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  3. 2021Palette: Image-to-Image Diffusion Models10 Nov
  4. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
24 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  12. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  13. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  14. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  15. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  16. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  17. 2021Paint by Wordcited by 107
  18. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  19. 2021GAN-Control: Explicitly Controllable GANscited by 91
  20. 2022A very preliminary analysis of DALL-E 2cited by 82
  21. 2021Learning to Compose Visual Relationscited by 72
  22. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  23. 2021Controllable and Compositional Generation with Latent-Space Energy-Based Modelscited by 64
  24. 2018ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributescited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 173; 2024: 195; 2025: 136; 2026: 94.

619 papers.

Sort:

By area:

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  4. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  5. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  7. 2022Diffusion Models in Vision: A Surveycited by 1,008
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  10. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  11. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  12. 2023Erasing Concepts from Diffusion Modelscited by 530
38 more
  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  3. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  4. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  6. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  7. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  8. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  9. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  10. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  11. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  12. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  13. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  14. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  15. 2024Guiding a Diffusion Model with a Bad Version of Itselfcited by 286
  16. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  17. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  18. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  19. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  20. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  21. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  22. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  23. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  24. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  25. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  26. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 187
  27. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  28. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  29. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  30. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  31. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 169
  32. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  33. 2025History-Guided Video Diffusioncited by 165
  34. 2023One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applicationscited by 164
  35. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  36. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  37. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  38. 2023Video Language Planningcited by 160

50 most cited of 619 All 619 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds
  2. 2024Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models
  3. 2022Implementing and Experimenting with Diffusion Models for Text-to-Image Generation
  4. 2025Emergence and Evolution of Interpretable Concepts in Diffusion Models
  5. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
  6. 2024Generating Intermediate Representations for Compositional Text-To-Image Generation
  7. 2026When Do Diffusion Models learn to Generate Multiple Objects?
  8. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Models