mlsanityData to 7 Oct 2026

Localizing Object-level Shape Variations with Text-to-Image Diffusion Models

arXiv 2303.11306 ↗First posted 20 Mar 2023cs.CV cs.GR cs.LGcited by 142

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image models give rise to workflows which often begin with an exploration step, where users sift through a large collection of generated images. The global nature of the text-to-image generation process prevents users from narrowing their exploration to a particular object in the image. In this paper, we present a technique to generate a collection of images that depicts variations in the shape of a specific object, enabling an object-level shape exploration process. Creating plausible variations is challenging as it requires control over the shape of the generated object while respecting its semantics. A particular challenge when generating object variations is accurately localizing the manipulation applied over the object's shape. We introduce a prompt-mixing technique that switches between prompts along the denoising process to attain a variety of shape choices. To localize the image-space operation, we present two techniques that use the self-attention layers in conjunction with the cross-attention layers. Moreover, we show that these localization techniques are general and effective beyond the scope of generating object variations. Extensive results and comparisons demonstrate the effectiveness of our method in generating object variations, and the competence of our localization techniques.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

51 of 59 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  4. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  5. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  6. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  8. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  9. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  10. 2022MagicMix: Semantic Mixing with Diffusion Models28 Oct
  11. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  12. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
6 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  2. 2022Blended Latent Diffusion6 Jun
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  6. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
21 more
  1. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  4. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2021Image Segmentation Using Text and Image Promptscited by 509
  10. 2023Zero-shot Image-to-Image Translationcited by 503
  11. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  14. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  15. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  16. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  17. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  18. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  19. 2020Editing in Style: Uncovering the Local Semantics of GANscited by 183
  20. 2016Learning What and Where to Drawcited by 105
  21. 2022Multiresolution Textual Inversioncited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 29; 2024: 60; 2025: 35; 2026: 18.

142 papers.

Sort:

By area:

  1. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  2. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  3. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  4. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  5. 2024ReNoise: Real Image Inversion Through Iterative Noisingcited by 127
  6. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  7. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  8. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  9. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  10. 2024One-Step Image Translation with Text-to-Image Modelscited by 111
  11. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 111
  12. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
38 more
  1. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 89
  2. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  3. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editingcited by 76
  4. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 68
  5. 2024TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Modelscited by 64
  6. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  7. 2024Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Modelscited by 62
  8. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 61
  9. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  10. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  11. 2023On Exact Inversion of DPM-Solverscited by 47
  12. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Modelscited by 47
  13. 2023Create Your World: Lifelong Text-to-Image Diffusioncited by 46
  14. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 42
  15. 2023GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videoscited by 39
  16. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  17. 2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38
  18. 2024Attention-Driven Training-Free Efficiency Enhancement of Diffusion Modelscited by 37
  19. 2023SLiMe: Segment Like Mecited by 34
  20. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  21. 2024Attention Calibration for Disentangled Text-to-Image Personalizationcited by 31
  22. 2024Motion Inversion for Video Customizationcited by 30
  23. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  24. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 29
  25. 2024PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradationcited by 29
  26. 2023LIME: Localized Image Editing via Attention Regularization in Diffusion Modelscited by 27
  27. 2024FreeTuner: Any Subject in Any Style with Training-free Diffusioncited by 24
  28. 2024DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenescited by 24
  29. 2023Text-Guided Synthesis of Eulerian Cinemagraphscited by 22
  30. 2024DiffUHaul: A Training-Free Method for Object Dragging in Imagescited by 22
  31. 2023CLiC: Concept Learning in Contextcited by 20
  32. 2024PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Controlcited by 20
  33. 2025Emergence and Evolution of Interpretable Concepts in Diffusion Modelscited by 20
  34. 2024Consolidating Attention Features for Multi-view Image Editingcited by 19
  35. 2024Lazy Diffusion Transformer for Interactive Image Editingcited by 19
  36. 2024Prompt-guided Precise Audio Editing with Diffusion Modelscited by 18
  37. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  38. 2023PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editorcited by 17

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Shape-Guided Diffusion with Inside-Outside Attention
  2. 2024LoMOE: Localized Multi-Object Editing via Multi-Diffusion
  3. 2024Improving Text-guided Object Inpainting with Semantic Pre-inpainting
  4. 2025Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models
  5. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model
  6. 2025DEAL-300K: Diffusion-based Editing Area Localization with a 300K-Scale Dataset and Frequency-Prompted Baseline
  7. 2024ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images
  8. 2023Local Conditional Controlling for Text-to-Image Diffusion Models