mlsanityData to 7 Oct 2026

InstanceDiffusion: Instance-level Control for Image Generation

arXiv 2402.03290 ↗First posted 5 Feb 2024cs.CV cs.AI cs.LGcited by 175

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image. We introduce InstanceDiffusion that adds precise instance-level control to text-to-image diffusion models. InstanceDiffusion supports free-form language conditions per instance and allows flexible ways to specify instance locations such as simple single points, scribbles, bounding boxes or intricate instance segmentation masks, and combinations thereof. We propose three major changes to text-to-image models that enable precise instance-level control. Our UniFusion block enables instance-level conditions for text-to-image models, the ScaleU block improves image fidelity, and our Multi-instance Sampler improves generations for multiple instances. InstanceDiffusion significantly surpasses specialized state-of-the-art models for each location condition. Notably, on the COCO dataset, we outperform previous state-of-the-art by 20.4% AP50box for box inputs, and 25.4% IoU for mask inputs.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 59 of its 70 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

59 of 70 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  2. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  3. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  6. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  7. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  8. 2022Shape-Guided Diffusion with Inside-Outside Attention1 Dec
  9. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
1 more
  1. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Segment Anythingcited by 8,158
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
34 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  13. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  14. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  15. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  16. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  17. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  18. 2016Generative Adversarial Text to Image Synthesiscited by 735
  19. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  20. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  21. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  22. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2018Video-to-Video Synthesiscited by 388
  25. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  26. 2016ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentationcited by 330
  27. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  28. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  29. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  30. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  31. 2021Paint by Wordcited by 107
  32. 2022X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusioncited by 49
  33. 2018Interactive Full Image Segmentation by Considering All Regions Jointlycited by 34
  34. 2020Conditional Image Generation and Manipulation for User-Specified Contentcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 49; 2025: 87; 2026: 39.

175 papers.

Sort:

By area:

  1. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  2. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  3. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  4. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  5. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  6. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  7. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42
  8. 2024HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generationcited by 39
  9. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  10. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  11. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  12. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
38 more
  1. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  2. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  3. 2024AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generationcited by 32
  4. 2025EliGen: Entity-Level Controlled Image Generation with Regional Attentioncited by 32
  5. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  6. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  7. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  8. 2025DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Modelscited by 28
  9. 2024Training-free Regional Prompting for Diffusion Transformerscited by 26
  10. 2025UniCombine: Unified Multi-Conditional Combination with Diffusion Transformercited by 25
  11. 2024ZeST: Zero-Shot Material Transfer from a Single Imagecited by 24
  12. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  13. 2024AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generationcited by 20
  14. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  15. 2024IFAdapter: Instance Feature Control for Grounded Text-to-Image Generationcited by 18
  16. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  17. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 18
  18. 2024ODGEN: Domain-specific Object Detection Data Generation with Diffusion Modelscited by 17
  19. 2025DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Modecited by 17
  20. 2024DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generationcited by 16
  21. 2025AnyStory: Towards Unified Single and Multiple Subject Personalization in Text-to-Image Generationcited by 16
  22. 2025Object-level Visual Prompts for Compositional Image Generationcited by 14
  23. 2024PartCraft: Crafting Creative Objects by Partscited by 13
  24. 2024GroundingBooth: Grounding Text-to-Image Customizationcited by 13
  25. 2025EarthSynth: Generating Informative Earth Observation with Diffusion Modelscited by 12
  26. 2024VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesiscited by 11
  27. 2024ID-Patch: Robust ID Association for Group Photo Personalizationcited by 11
  28. 2025OmniPSD: Layered PSD Generation with Diffusion Transformercited by 11
  29. 2025DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Modelscited by 10
  30. 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generationcited by 10
  31. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  32. 2024ROICtrl: Boosting Instance Control for Visual Generationcited by 9
  33. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  34. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 9
  35. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 9
  36. 2025SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulationcited by 9
  37. 2024SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Modelscited by 8
  38. 2025ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoningcited by 8

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TRACE: Your Diffusion Model is Secretly an Instance Edge Detector
  2. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis
  3. 2026InstanceControl: Controllable Complex Image Generation without Instance Labeling
  4. 2024CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
  5. 2026InstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning
  6. 2023Local Conditional Controlling for Text-to-Image Diffusion Models
  7. 2023Diffusion Self-Guidance for Controllable Image Generation
  8. 2024LocRef-Diffusion:Tuning-Free Layout and Appearance-Guided Generation