# On Conditioning the Input Noise for Controlled Image Generation with Diffusion Models

arXiv: https://arxiv.org/abs/2205.03859 · HTML: https://mlsanity.com/paper/2205.03859/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 8 May 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 18 of 18)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](https://mlsanity.com/paper/2112.05744/) (2021)
- [EditGAN: High-Precision Semantic Image Editing](https://mlsanity.com/paper/2111.03186/) (2021)
- [Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing](https://mlsanity.com/paper/2104.07021/) (2021)
- [SieveNet: A Unified Framework for Robust Image-Based Virtual Try-On](https://mlsanity.com/paper/2001.06265/) (2020)
- [MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANs](https://mlsanity.com/paper/2012.02821/) (2020)

## Built on by (most cited first, 14 of 14)

- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models](https://mlsanity.com/paper/2212.00842/) (2022)
- [You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale](https://mlsanity.com/paper/2412.06699/) (2024)
- [Diffusion Posterior Proximal Sampling for Image Restoration](https://mlsanity.com/paper/2402.16907/) (2024)
- [Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Models](https://mlsanity.com/paper/2312.11473/) (2023)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [Enhancing Diffusion Model Stability for Image Restoration via Gradient Management](https://mlsanity.com/paper/2507.06656/) (2025)
- [A Comparative Study of Variational Autoencoders, Normalizing Flows, and Score-based Diffusion Models for Electrical Impedance Tomography](https://mlsanity.com/paper/2310.15831/) (2023)
- [Conditional Diffusion with Less Explicit Guidance via Model Predictive Control](https://mlsanity.com/paper/2210.12192/) (2022)
- [AortaDiff: Volume-Guided Conditional Diffusion Models for Multi-Branch Aortic Surface Generation](https://mlsanity.com/paper/2507.13404/) (2025)
- [3D Structure-guided Network for Tooth Alignment in 2D Photograph](https://mlsanity.com/paper/2310.11106/) (2023)
- [Few-Shot Anomaly-Driven Generation for Anomaly Classification and Segmentation](https://mlsanity.com/paper/2505.09263/) (2025)
- [Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation](https://mlsanity.com/paper/2603.22509/) (2026)

## Abstract (arXiv)

Conditional image generation has paved the way for several breakthroughs in image editing, generating stock photos and 3-D object generation. This continues to be a significant area of interest with the rise of new state-of-the-art methods that are based on diffusion models. However, diffusion models provide very little control over the generated image, which led to subsequent works exploring techniques like classifier guidance, that provides a way to trade off diversity with fidelity. In this work, we explore techniques to condition diffusion models with carefully crafted input noise artifacts. This allows generation of images conditioned on semantic attributes. This is different from existing approaches that input Gaussian noise and further introduce conditioning at the diffusion model's inference step. Our experiments over several examples and conditional settings show the potential of our approach.