mlsanityData to 14 Oct 2026

On Conditioning the Input Noise for Controlled Image Generation with Diffusion Models

arXiv 2205.03859 ↗First posted 8 May 2022cs.CV cs.LGcited by 14

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Conditional image generation has paved the way for several breakthroughs in image editing, generating stock photos and 3-D object generation. This continues to be a significant area of interest with the rise of new state-of-the-art methods that are based on diffusion models. However, diffusion models provide very little control over the generated image, which led to subsequent works exploring techniques like classifier guidance, that provides a way to trade off diversity with fidelity. In this work, we explore techniques to condition diffusion models with carefully crafted input noise artifacts. This allows generation of images conditioned on semantic attributes. This is different from existing approaches that input Gaussian noise and further introduce conditioning at the diffusion model's inference step. Our experiments over several examples and conditional settings show the potential of our approach.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 18 of its 23 references that are in the map. First posted on arXiv 8 May 2022.

Builds on

18 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 17

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  8. 2014Conditional Generative Adversarial Netscited by 2,162
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  11. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  12. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
5 more
  1. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  2. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  3. 2021Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editingcited by 54
  4. 2020SieveNet: A Unified Framework for Robust Image-Based Virtual Try-Oncited by 23
  5. 2020MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 2; 2025: 4; 2026: 1.

14 papers.

Sort:

By area:

  1. 2022Diffusion Models in Vision: A Surveycited by 1,015
  2. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 109
  3. 20223D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Modelscited by 67
  4. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  5. 2024Diffusion Posterior Proximal Sampling for Image Restorationcited by 14
  6. 2023Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Modelscited by 9
  7. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  8. 2025Enhancing Diffusion Model Stability for Image Restoration via Gradient Managementcited by 5
  9. 2023A Comparative Study of Variational Autoencoders, Normalizing Flows, and Score-based Diffusion Models for Electrical Impedance Tomographycited by 4
  10. 2022Conditional Diffusion with Less Explicit Guidance via Model Predictive Controlcited by 1
  11. 2025AortaDiff: Volume-Guided Conditional Diffusion Models for Multi-Branch Aortic Surface Generationcited by 1
  12. 20233D Structure-guided Network for Tooth Alignment in 2D Photographcited by 0
2 more
  1. 2025Few-Shot Anomaly-Driven Generation for Anomaly Classification and Segmentationcited by 0
  2. 2026Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Controllable Image Generation through Representation-Conditioned Diffusion Models
  2. 2024Conditional Image Synthesis with Diffusion Models: A Survey
  3. 2024A Simple Approach to Unifying Diffusion-based Conditional Generation
  4. 2023Conditional Generation from Unconditional Diffusion Models using Denoiser Representations
  5. 2025Guidance Free Image Editing via Explicit Conditioning
  6. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
  7. 2024Controllable Generation with Text-to-Image Diffusion Models: A Survey
  8. 2024Diffusion Model-Based Image Editing: A Survey