mlsanityData to 7 Oct 2026

Shape-Guided Diffusion with Inside-Outside Attention

arXiv 2212.00210 ↗First posted 1 Dec 2022cs.CV cs.AI cs.LGcited by 45

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion. Our training-free method uses an Inside-Outside Attention mechanism during the inversion and generation process to apply a shape constraint to the cross- and self-attention maps. Our mechanism designates which spatial region is the object (inside) vs. background (outside) then associates edits to the correct region. We demonstrate the efficacy of our method on the shape-guided editing task, where the model must replace an object according to a text prompt and object mask. We curate a new ShapePrompts benchmark derived from MS-COCO and achieve SOTA results in shape faithfulness without a degradation in text alignment or image realism according to both automatic metrics and annotator ratings. Our data and code will be made available at https://shape-guided-diffusion.github.io.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 40 of its 48 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

40 of 48 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  2. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  3. 2022UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image17 Oct
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  9. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
19 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  5. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  6. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  11. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  12. 2018Generative Image Inpainting with Contextual Attentioncited by 578
  13. 2021Image Segmentation Using Text and Image Promptscited by 509
  14. 2018Image Inpainting for Irregular Holes Using Partial Convolutionscited by 505
  15. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  16. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  17. 2021AP-10K: A Benchmark for Animal Pose Estimation in the Wildcited by 123
  18. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  19. 2022Shape-guided Object Inpaintingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 15; 2025: 9; 2026: 3.

45 papers.

Sort:

By area:

  1. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  2. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  3. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  4. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  5. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  6. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editingcited by 134
  7. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  8. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  9. 2023LLM-grounded Video Diffusion Modelscited by 83
  10. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Modelscited by 82
  11. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidancecited by 69
  12. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
33 more
  1. 2023Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Modelcited by 60
  2. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  3. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesiscited by 51
  4. 2024Attention Calibration for Disentangled Text-to-Image Personalizationcited by 31
  5. 2023DP-LDMs: Differentially Private Latent Diffusion Modelscited by 28
  6. 2023MDP: A Generalized Framework for Text-Guided Image Editing by Manipulating the Diffusion Pathcited by 18
  7. 2023PRedItOR: Text Guided Image Editing with Diffusion Priorcited by 16
  8. 2023Predicated Diffusion: Predicate Logic-Based Attention Guidance for Text-to-Image Diffusion Modelscited by 16
  9. 2023Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generationcited by 12
  10. 2024Text Guided Image Editing with Automatic Concept Locating and Forgettingcited by 12
  11. 2024FontStudio: Shape-Adaptive Diffusion Model for Coherent and Consistent Font Effect Generationcited by 9
  12. 2024XReal: Realistic Anatomy and Pathology-Aware X-ray Generation via Controllable Diffusion Modelcited by 8
  13. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8
  14. 2024Sketch-guided Image Inpainting with Partial Discrete Diffusion Processcited by 7
  15. 2025Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusioncited by 6
  16. 2026SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generationcited by 6
  17. 2023DIFF-NST: Diffusion Interleaving For deFormable Neural Style Transfercited by 5
  18. 2024InsertDiffusion: Identity Preserving Visualization of Objects through a Training-Free Diffusion Architecturecited by 5
  19. 2024A Survey of Multimodal Composite Editing and Retrievalcited by 5
  20. 2025Geometrical Properties of Text Token Embeddings for Strong Semantic Binding in Text-to-Image Generationcited by 5
  21. 2025Attention in Diffusion Model: A Surveycited by 4
  22. 2025POEM: Precise Object-level Editing via MLLM controlcited by 4
  23. 2025Exploring the latent space of diffusion models directly through singular value decompositioncited by 3
  24. 2024Scribble-Guided Diffusion for Training-free Text-to-Image Generationcited by 2
  25. 2025Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervisioncited by 2
  26. 2024Background-Aware Defect Generation for Robust Industrial Anomaly Detectioncited by 1
  27. 2025Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulationcited by 1
  28. 2024TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translationcited by 0
  29. 2024Enabling Region-Specific Control via Lassos in Point-Based Colorizationcited by 0
  30. 2025ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Modelscited by 0
  31. 2025MFGDiffusion: Mask-Guided Smoke Synthesis for Enhanced Forest Fire Detectioncited by 0
  32. 2026LAMS-Edit: Latent and Attention Mixing with Schedulers for Improved Content Preservation in Diffusion-Based Image and Style Editingcited by 0
  33. 2026Dynamic Frequency Modulation for Controllable Text-driven Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Models
  2. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model
  3. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  4. 2024Enhancing Image Layout Control with Loss-Guided Diffusion Models
  5. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  6. 2024MFTF: Mask-free Training-free Object Level Layout Control Diffusion Model
  7. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation
  8. 2024Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models