# Conditional Image Synthesis with Diffusion Models: A Survey

arXiv: https://arxiv.org/abs/2409.19365 · HTML: https://mlsanity.com/paper/2409.19365/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 239 of its 257 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 239)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)

## Built on by (most cited first, 29 of 29)

- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Feedback Guidance of Diffusion Models](https://mlsanity.com/paper/2506.06085/) (2025)
- [FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusion](https://mlsanity.com/paper/2502.05606/) (2025)
- [Image Editing with Diffusion Models: A Survey](https://mlsanity.com/paper/2504.13226/) (2025)
- [Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesis](https://mlsanity.com/paper/2506.00433/) (2025)
- [Veila: Panoramic LiDAR Generation from a Monocular RGB Image](https://mlsanity.com/paper/2508.03690/) (2025)
- [Learning Individual Behavior in Agent-Based Models with Graph Diffusion Networks](https://mlsanity.com/paper/2505.21426/) (2025)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [Adaptive Weighted h-Transform Sampling for Coarse-Guided Visual Generation](https://mlsanity.com/paper/2603.12057/) (2026)
- [Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving](https://mlsanity.com/paper/2605.22809/) (2026)
- [Prototype-Guided Diffusion for Digital Pathology: Achieving Foundation Model Performance with Minimal Clinical Data](https://mlsanity.com/paper/2504.12351/) (2025)
- [Generative Latent Diffusion for Efficient Spatiotemporal Data Reduction](https://mlsanity.com/paper/2507.02129/) (2025)
- [Minimum Data, Maximum Impact: 20 annotated samples for explainable lung nodule classification](https://mlsanity.com/paper/2508.00639/) (2025)
- [q3-MuPa: Quick, Quiet, Quantitative Multi-Parametric MRI using Physics-Informed Diffusion Models](https://mlsanity.com/paper/2512.23726/) (2025)
- [DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion](https://mlsanity.com/paper/2602.22610/) (2026)
- [Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance](https://mlsanity.com/paper/2603.20584/) (2026)
- [Reducing Experimental Testing in Space Propulsion Film Cooling Analyses by Pixelwise Generative Image Interpolation](https://mlsanity.com/paper/2605.29911/) (2026)
- [Treatment-Conditioned Diffusion for Forecasting Neurodegenerative Disease Progression](https://mlsanity.com/paper/2605.29932/) (2026)
- [Generative diffusion model surrogates for mechanistic agent-based biological models](https://mlsanity.com/paper/2505.09630/) (2025)
- [Enhancing Diffusion Face Generation with Contrastive Embeddings and SegFormer Guidance](https://mlsanity.com/paper/2508.09847/) (2025)
- [One-shot Conditional Sampling: MMD meets Nearest Neighbors](https://mlsanity.com/paper/2509.25507/) (2025)
- [Signal-Aware Conditional Diffusion Surrogates for Transonic Wing Pressure Prediction](https://mlsanity.com/paper/2604.11263/) (2026)
- [Optimally Bridging Semantics and Data: Generative Semantic Communication via Schr"odinger Bridge](https://mlsanity.com/paper/2604.17802/) (2026)
- [Landscape-Awareness for Geometric View Diffusion Model](https://mlsanity.com/paper/2605.19865/) (2026)
- [HITL-D: Human In The Loop Diffusion Assisted Shared Control](https://mlsanity.com/paper/2605.21460/) (2026)
- [Spectral Guidance for Flexible and Efficient Control of Diffusion Models](https://mlsanity.com/paper/2605.28900/) (2026)
- [Conditioning Matters: Stabilizing Inversion and Attention in Diffusion Image Editing](https://mlsanity.com/paper/2606.14125/) (2026)
- [Appearance Pointers -- Multimodal Region Control of Diffusion Transformers](https://mlsanity.com/paper/2607.19344/) (2026)
- [Generative AI-Based Data Augmentation for Oral Lesion Classification: The PhotoMOCI Dataset and Benchmark](https://mlsanity.com/paper/2609.35226/) (2026)

## Abstract (arXiv)

Conditional image synthesis based on user-specified requirements is a key component in creating complex visual content. In recent years, diffusion-based generative modeling has become a highly effective way for conditional image synthesis, leading to exponential growth in the literature. However, the complexity of diffusion-based modeling, the wide range of image synthesis tasks, and the diversity of conditioning mechanisms present significant challenges for researchers to keep up with rapid developments and to understand the core concepts on this topic. In this survey, we categorize existing works based on how conditions are integrated into the two fundamental components of diffusion-based modeling, textiti.e., the denoising network and the sampling process. We specifically highlight the underlying principles, advantages, and potential challenges of various conditioning approaches during the training, re-purposing, and specialization stages to construct a desired denoising network. We also summarize six mainstream conditioning mechanisms in the sampling process. All discussions are centered around popular applications. Finally, we pinpoint several critical yet still unsolved problems and suggest some possible solutions for future research. Our reviewed works are itemized at https://github.com/zju-pi/Awesome-Conditional-Diffusion-Models.