mlsanityData to 14 Oct 2026

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models

arXiv 2108.02938 ↗First posted 6 Aug 2021cs.CVcited by 635

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Denoising diffusion probabilistic models (DDPM) have shown remarkable performance in unconditional image generation. However, due to the stochasticity of the generative process in DDPM, it is challenging to generate images with the desired semantics. In this work, we propose Iterative Latent Variable Refinement (ILVR), a method to guide the generative process in DDPM to generate high-quality images based on a given reference image. Here, the refinement of the generative process in DDPM enables a single DDPM to sample images from various sets directed by the reference image. The proposed ILVR method generates high-quality images while controlling the generation. The controllability of our method allows adaptation of a single DDPM without any additional learning in various image generation tasks, such as generation from various downsampling factors, multi-domain image translation, paint-to-image, and editing with scribbles.

Key facts. Cited by 635 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 6 Aug 2021.

Builds on

51 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  2. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  3. 2020Denoising Diffusion Probabilistic Models19 Jun
  4. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  9. 2016Wide Residual Networkscited by 3,070
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  11. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  12. 2018Group Normalizationcited by 1,524
35 more
  1. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  2. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  5. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  6. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  7. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  9. 2018Self-Attention Generative Adversarial Networkscited by 809
  10. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  12. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  13. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 695
  14. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  15. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 639
  16. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  17. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  18. 2020GANSpace: Discovering Interpretable GAN Controlscited by 614
  19. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  20. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  21. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  22. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  23. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  24. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  25. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  26. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253
  27. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  28. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  29. 2019Image Processing Using Multi-Code GAN Priorcited by 185
  30. 2019Photorealistic Style Transfer via Wavelet Transformscited by 166
  31. 2016Semantic Image Inpainting with Deep Generative Modelscited by 157
  32. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  33. 2020Transforming and Projecting Images into Class-conditional Generative Networkscited by 76
  34. 2020Semantic Pyramid for Image Generationcited by 34
  35. 2021Toward Spatially Unbiased Generative Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 93; 2023: 195; 2024: 171; 2025: 112; 2026: 53.

635 papers.

Sort:

By area:

  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  5. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,416
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  8. 2022Diffusion Models in Vision: A Surveycited by 1,015
  9. 2022Denoising Diffusion Restoration Modelscited by 963
  10. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  11. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  12. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
38 more
  1. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 583
  2. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  3. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 508
  4. 2023Zero-shot Image-to-Image Translationcited by 505
  5. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  6. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 489
  7. 2022Diffusion Models for Adversarial Purificationcited by 476
  8. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  9. 2022Blended Latent Diffusioncited by 432
  10. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  11. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  12. 2023ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shiftingcited by 365
  13. 2023DeepCache: Accelerating Diffusion Models for Freecited by 359
  14. 2023Human Motion Diffusion as a Generative Priorcited by 357
  15. 2023Denoising Diffusion Models for Plug-and-Play Image Restorationcited by 340
  16. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 334
  17. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  18. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  19. 2021Score-based diffusion models for accelerated MRIcited by 311
  20. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 307
  21. 2022Perception Prioritized Training of Diffusion Modelscited by 303
  22. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 294
  23. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 291
  24. 2024Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wildcited by 286
  25. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  26. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 280
  27. 2022A Survey on Generative Diffusion Modelcited by 240
  28. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 238
  29. 2023SinSR: Diffusion-Based Image Super-Resolution in a Single Stepcited by 238
  30. 2022Diffusion Models for Medical Anomaly Detectioncited by 232
  31. 2022Diffusion models as plug-and-play priorscited by 231
  32. 2022BBDM: Image-to-image Translation with Brownian Bridge Diffusion Modelscited by 225
  33. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 223
  34. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Modelscited by 222
  35. 2022Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Modelscited by 215
  36. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 213
  37. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 213
  38. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 212

50 most cited of 635 All 635 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Denoising Diffusion Implicit Models
  2. 2024Enhancing Diffusion Models for High-Quality Image Generation
  3. 2022Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models
  4. 2023Improving Denoising Diffusion Probabilistic Models via Exploiting Shared Representations
  5. 2026Inverting the Generation Process of Denoising Diffusion Implicit Models: Empirical Evaluation and a Novel Method
  6. 2022Dynamic Dual-Output Diffusion Models
  7. 2025Compressed Image Generation with Denoising Diffusion Codebook Models
  8. 2024I-Diff: Structural Regularization for High-Fidelity Diffusion Models