mlsanityData to 14 Oct 2026

Diffusion Self-Guidance for Controllable Image Generation

arXiv 2306.00986 ↗First posted 1 Jun 2023cs.CV cs.LG stat.MLcited by 305

Not on the map. Why?

Cited by 187 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale generative models are capable of producing high-quality images from detailed text descriptions. However, many aspects of an image are difficult or impossible to convey through text. We introduce self-guidance, a method that provides greater control over generated images by guiding the internal representations of diffusion models. We demonstrate that properties such as the shape, location, and appearance of objects can be extracted from these representations and used to steer sampling. Self-guidance works similarly to classifier guidance, but uses signals present in the pretrained model itself, requiring no additional models or training. We show how a simple set of properties can be composed to perform challenging image manipulations, such as modifying the position or size of objects, merging the appearance of objects in one image with the layout of another, composing objects from many images into one, and more. We also show that self-guidance can be used to edit real images. For results and an interactive demo, see our project page at https://dave.ml/selfguidance/

Key facts. Cited by 305 papers in the map (data through 14 Oct 2026). Builds on 40 of its 40 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

40 of 40 references.

References · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
28 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  8. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  12. 2021Variational Diffusion Modelscited by 1,153
  13. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  14. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  15. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  16. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  17. 2023Universal Guidance for Diffusion Modelscited by 437
  18. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  19. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  20. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  21. 2022Diffusion models as plug-and-play priorscited by 231
  22. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  23. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  24. 2021Unsupervised Discovery of Object Radiance Fieldscited by 101
  25. 2021Using latent space regression to analyze and leverage compositionality in GANscited by 55
  26. 2022BlobGAN: Spatially Disentangled Scene Representationscited by 35
  27. 2022Region-Based Semantic Factorization in GANscited by 22
  28. 2022LECA: A Learned Approach for Efficient Cover-agnostic Watermarkingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 40; 2024: 135; 2025: 100; 2026: 30.

305 papers.

Sort:

By area:

  1. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  2. 2023Style Aligned Image Generation via Shared Attentioncited by 216
  3. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  4. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
  5. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  6. 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 143
  7. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  8. 2024Implicit Style-Content Separation using B-LoRAcited by 135
  9. 2024ReNoise: Real Image Inversion Through Iterative Noisingcited by 128
  10. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  11. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  12. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
38 more
  1. 2024GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editingcited by 115
  2. 2023Self-correcting LLM-controlled Diffusion Modelscited by 113
  3. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 113
  4. 2023Generative Image Dynamicscited by 94
  5. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 94
  6. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 91
  7. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 90
  8. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  9. 2023LLM-grounded Video Diffusion Modelscited by 83
  10. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  11. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 69
  12. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  13. 2023HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Modelscited by 67
  14. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 67
  15. 2023OBJECT 3DIT: Language-guided 3D-aware Image Editingcited by 63
  16. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  17. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  18. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  19. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
  20. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  21. 2024DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalizationcited by 60
  22. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3Dcited by 59
  23. 2024FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesiscited by 59
  24. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 59
  25. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 58
  26. 2023EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methodscited by 54
  27. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 54
  28. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  29. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  30. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  31. 2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 52
  32. 2024MotionCraft: Physics-based Zero-Shot Video Generationcited by 48
  33. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 48
  34. 2024Stable-Makeup: When Real-World Makeup Transfer Meets Diffusion Modelcited by 45
  35. 2024SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teachercited by 45
  36. 2024Flow Matching with Gaussian Process Priors for Probabilistic Time Series Forecastingcited by 43
  37. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  38. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 41

50 most cited of 305 All 305 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Self-Guided Diffusion Models
  2. 2026Towards Controllable Image Generation through Representation-Conditioned Diffusion Models
  3. 2024Diffusion Self-Distillation for Zero-Shot Customized Image Generation
  4. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  5. 2023Filter-Guided Diffusion for Controllable Image Generation
  6. 2024Self-Guidance: Boosting Flow and Diffusion Generation on Their Own
  7. 2023SEGA: Instructing Text-to-Image Models using Semantic Guidance
  8. 2023Generate Anything Anywhere in Any Scene