mlsanityData to 7 Oct 2026

Guided Image Synthesis via Initial Image Editing in Diffusion Model

arXiv 2305.03382 ↗First posted 5 May 2023cs.CVcited by 63

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have the ability to generate high quality images by denoising pure Gaussian noise images. While previous research has primarily focused on improving the control of image generation through adjusting the denoising process, we propose a novel direction of manipulating the initial noise to control the generated image. Through experiments on stable diffusion, we show that blocks of pixels in the initial latent images have a preference for generating specific content, and that modifying these blocks can significantly influence the generated image. In particular, we show that modifying a part of the initial image affects the corresponding region of the generated image while leaving other regions unaffected, which is useful for repainting tasks. Furthermore, we find that the generation preferences of pixel blocks are primarily determined by their values, rather than their position. By moving pixel blocks with a tendency to generate user-desired content to user-specified regions, our approach achieves state-of-the-art performance in layout-to-image generation. Our results highlight the flexibility and power of initial image manipulation in controlling the generated image. Project Page: https://ut-mao.github.io/swap.github.io/

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 30 of its 33 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

30 of 33 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  4. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  5. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  7. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  8. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  9. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  12. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
6 more
  1. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  2. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  3. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  4. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  5. 2021You Only Learn One Representation: Unified Network for Multiple Taskscited by 86
  6. 2023Training-Free Location-Aware Text-to-Image Synthesiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 28; 2025: 15; 2026: 12.

63 papers.

Sort:

By area:

  1. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  2. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  3. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  4. 2024Training-free Camera Control for Video Generationcited by 119
  5. 2024Not All Noises Are Created Equally:Diffusion Noise Selection and Optimizationcited by 70
  6. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 59
  7. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 47
  8. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  9. 2023Interpretable Diffusion via Information Decompositioncited by 37
  10. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  11. 2024Identity Decoupling for Multi-Subject Personalization of Text-to-Image Modelscited by 33
  12. 2024StableDrag: Stable Dragging for Point-based Image Editingcited by 28
38 more
  1. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  2. 2024NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Mergingcited by 25
  3. 2023The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initializationcited by 22
  4. 2024LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videoscited by 18
  5. 2024The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noisecited by 18
  6. 2023InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Advisercited by 17
  7. 2025MangaNinja: Line Art Colorization with Precise Reference Followingcited by 17
  8. 2024Spatial-Aware Latent Initialization for Controllable Image Generationcited by 15
  9. 2024Taming Stable Diffusion for Text to 360deg Panorama Image Generationcited by 12
  10. 2024TKG-DM: Training-free Chroma Key Content Generation Diffusion Modelcited by 11
  11. 2024MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Modelscited by 11
  12. 2023High-Fidelity Diffusion-based Image Editingcited by 10
  13. 2024Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editingcited by 10
  14. 2023Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of Latent-Based Diffusion Modelscited by 9
  15. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  16. 2024Training-Free Sketch-Guided Diffusion with Latent Optimizationcited by 7
  17. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Modelscited by 7
  18. 2025Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinementcited by 7
  19. 2024Isolated Diffusion: Optimizing Multi-Concept Text-to-Image Generation Training-Freely with Isolated Diffusion Guidancecited by 6
  20. 2024Generating Coherent Sequences of Visual Illustrations for Real-World Manual Taskscited by 6
  21. 2024SynFER: Towards Boosting Facial Expression Recognition with Synthetic Datacited by 6
  22. 2025Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editingcited by 5
  23. 2025TiP4GEN: Text to Immersive Panorama 4D Scene Generationcited by 5
  24. 2024A Comprehensive Survey on Diffusion Models and Their Applicationscited by 4
  25. 2024Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conceptioncited by 3
  26. 2024Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detectioncited by 3
  27. 2024Leapfrog Latent Consistency Model (LLCM) for Medical Images Generationcited by 3
  28. 2025NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaborationcited by 3
  29. 2025LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editingcited by 3
  30. 2024Difficulty Controlled Diffusion Model for Synthesizing Effective Training Datacited by 2
  31. 2024Label-Efficient Data Augmentation with Video Diffusion Models for Guidewire Segmentation in Cardiac Fluoroscopycited by 2
  32. 2025Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptationcited by 2
  33. 2025Noise-Level Diffusion Guidance: Well Begun is Half Donecited by 2
  34. 2025Demystifying Numerosity in Diffusion Models -- Limitations and Remediescited by 2
  35. 2025TAUE: Training-free Noise Transplant and Cultivation Diffusion Modelcited by 2
  36. 2026COT-FM: Cluster-wise Optimal Transport Flow Matchingcited by 2
  37. 2026SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generationcited by 2
  38. 2024StyleInject: Parameter Efficient Tuning of Text-to-Image Diffusion Modelscited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Differential Diffusion: Giving Each Pixel Its Strength
  2. 2024Diffusion Model-Based Image Editing: A Survey
  3. 2024Enhancing Image Layout Control with Loss-Guided Diffusion Models
  4. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation
  5. 2022High-Fidelity Guided Image Synthesis with Latent Diffusion Models
  6. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  7. 2024Conditional Image Synthesis with Diffusion Models: A Survey
  8. 2023Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later