mlsanityData to 7 Oct 2026

A Diffusion-based Method for Multi-turn Compositional Image Generation

arXiv 2304.02192 ↗First posted 5 Apr 2023cs.CV cs.AI cs.LGcited by 4

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Multi-turn compositional image generation (M-CIG) is a challenging task that aims to iteratively manipulate a reference image given a modification text. While most of the existing methods for M-CIG are based on generative adversarial networks (GANs), recent advances in image generation have demonstrated the superiority of diffusion models over GANs. In this paper, we propose a diffusion-based method for M-CIG named conditional denoising diffusion with image compositional matching (CDD-ICM). We leverage CLIP as the backbone of image and text encoders, and incorporate a gated fusion mechanism, originally proposed for question answering, to compositionally fuse the reference image and the modification text at each turn of M-CIG. We introduce a conditioning scheme to generate the target image based on the fusion results. To prioritize the semantic quality of the generated target image, we learn an auxiliary image compositional match (ICM) objective, along with the conditional denoising diffusion (CDD) objective in a multi-task learning framework. Additionally, we also perform ICM guidance and classifier-free guidance to improve performance. Experimental results show that CDD-ICM achieves state-of-the-art results on two benchmark datasets for M-CIG, i.e., CoDraw and i-CLEVR.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 60 of its 71 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

60 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  5. 2021Palette: Image-to-Image Diffusion Models10 Nov
  6. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  11. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,250
  12. 2014Image Super-Resolution Using Deep Convolutional Networkscited by 1,218
  13. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  14. 2015Accurate Image Super-Resolution Using Very Deep Convolutional Networkscited by 902
  15. 2018Image Transformercited by 770
  16. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  17. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  18. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  19. 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 234
  20. 2020SRFlow: Learning the Super-Resolution Space with Normalizing Flowcited by 209
  21. 2019EdgeConnect: Generative Image Inpainting with Adversarial Edge Learningcited by 161
  22. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  23. 2019StructureFlow: Image Inpainting via Structure-aware Appearance Flowcited by 118
  24. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  25. 2020Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizationscited by 102
  26. 2019Learning Pyramid-Context Encoder Network for High-Quality Image Inpaintingcited by 94
  27. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  28. 2018Shift-Net: Image Inpainting via Deep Feature Rearrangementcited by 87
  29. 2018Image Inpainting via Generative Multi-column Convolutional Neural Networkscited by 81
  30. 2018How good is my GAN?cited by 74
  31. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  32. 2018SPG-Net: Segmentation Prediction and Guidance Network for Image Inpaintingcited by 58
  33. 2017Generalizing Hamiltonian Monte Carlo with Neural Networkscited by 52
  34. 2017Be Your Own Prada: Fashion Synthesis with Structural Coherencecited by 49
  35. 2017Pixel Recursive Super Resolutioncited by 44
  36. 2013Deep Generative Stochastic Networks Trainable by Backpropcited by 36
  37. 2017A-NICE-MC: Adversarial Training for MCMCcited by 36
  38. 2020SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoningcited by 32

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generationcited by 18
  2. 2024Preference Adaptive and Sequential Text-to-Image Generationcited by 8
  3. 2023Training Priors Predict Text-To-Image Model Performancecited by 4
  4. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward
  2. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
  3. 2023Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models
  4. 2023Are Diffusion Models Vision-And-Language Reasoners?
  5. 2024Generating Intermediate Representations for Compositional Text-To-Image Generation
  6. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models
  7. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis
  8. 2025Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Models