mlsanityData to 7 Oct 2026

MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation

arXiv 2302.08113 ↗First posted 16 Feb 2023cs.CVcited by 529

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent advances in text-to-image generation with diffusion models present transformative capabilities in image quality. However, user controllability of the generated image, and fast adaptation to new tasks still remains an open challenge, currently mostly addressed by costly and long re-training and fine-tuning or ad-hoc adaptations to specific image generation tasks. In this work, we present MultiDiffusion, a unified framework that enables versatile and controllable image generation, using a pre-trained text-to-image diffusion model, without any further training or finetuning. At the center of our approach is a new generation process, based on an optimization task that binds together multiple diffusion generation processes with a shared set of parameters or constraints. We show that MultiDiffusion can be readily applied to generate high quality and diverse images that adhere to user-provided controls, such as desired aspect ratio (e.g., panorama), and spatial guiding signals, ranging from tight segmentation masks to bounding boxes. Project webpage: https://multidiffusion.github.io

Key facts. Cited by 529 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 50 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

46 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  3. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  4. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  6. 2022Diffusion-based Image Translation using Disentangled Style and Content Representation30 Sep
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  8. 2022Blended Latent Diffusion6 Jun
  9. 2022Pretraining is All You Need for Image-to-Image Translation25 May
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
3 more
  1. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  2. 2021Palette: Image-to-Image Diffusion Models10 Nov
  3. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
19 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2022Diffusion Models in Vision: A Surveycited by 1,008
  8. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  9. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  10. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  11. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  14. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  15. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  16. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  17. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  18. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  19. 2022Leveraging Off-the-shelf Diffusion Model for Multi-attribute Fashion Image Manipulationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 86; 2024: 194; 2025: 152; 2026: 97.

529 papers.

Sort:

By area:

  1. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  3. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  4. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  5. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  6. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  7. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  8. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  9. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  10. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  11. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  12. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
38 more
  1. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  2. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  3. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  4. 2023Teaching CLIP to Count to Tencited by 194
  5. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  6. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  7. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  8. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  9. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  10. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 135
  11. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  12. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 127
  13. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  14. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  15. 20234M: Massively Multimodal Masked Modelingcited by 118
  16. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  17. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 115
  18. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  19. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 108
  20. 2023DemoFusion: Democratising High-Resolution Image Generation With No cited by 105
  21. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  22. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  23. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  24. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  25. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  26. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  27. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 86
  28. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  29. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  30. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
  31. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 81
  32. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  33. 2025Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysiscited by 78
  34. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  35. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
  36. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  37. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  38. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74

50 most cited of 529 All 529 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-condition Image Synthesis
  2. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Model
  3. 2024Panoptic Diffusion Models: co-generation of images and segmentation maps
  4. 2024One Diffusion to Generate Them All
  5. 2026Improving Controllable Generation: Faster Training and Better Performance via x0-Supervision
  6. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
  7. 2023DiffCollage: Parallel Generation of Large Content with Diffusion Models
  8. 2025Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation