mlsanityData to 7 Oct 2026

Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation

arXiv 2303.09319 ↗First posted 16 Mar 2023cs.CVcited by 60

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Language-guided image generation has achieved great success nowadays by using diffusion models. However, texts can be less detailed to describe highly-specific subjects such as a particular dog or a certain car, which makes pure text-to-image generation not accurate enough to satisfy user requirements. In this work, we present a novel Unified Multi-Modal Latent Diffusion (UMM-Diffusion) which takes joint texts and images containing specified subjects as input sequences and generates customized images with the subjects. To be more specific, both input texts and images are encoded into one unified multi-modal latent space, in which the input images are learned to be projected to pseudo word embedding and can be further combined with text to guide image generation. Besides, to eliminate the irrelevant parts of the input images such as background or illumination, we propose a novel sampling technique of diffusion models used by the image generator which fuses the results guided by multi-modal input and pure text input. By leveraging the large-scale pre-trained text-to-image generator and the designed image encoder, our method is able to generate high-quality images with complex semantics from both aspects of input texts and images.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 40 of its 50 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

40 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
23 more
  1. 2015Fast R-CNNcited by 2,930
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2014Conditional Generative Adversarial Netscited by 2,152
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2016Generative Adversarial Text to Image Synthesiscited by 735
  8. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  9. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  10. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  11. 2019Controllable Text-to-Image Generationcited by 187
  12. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  13. 2019DoveNet: Deep Image Harmonization via Domain Verificationcited by 134
  14. 2017Deep Image Harmonizationcited by 120
  15. 2021Instance-Conditioned GANcited by 119
  16. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  17. 2018ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositingcited by 90
  18. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  19. 2019Improving the Harmony of the Composite Image by Spatial-Separated Attention Modulecited by 73
  20. 2017GP-GAN: Towards Realistic High-Resolution Image Blendingcited by 64
  21. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  22. 2021Towards Open-World Text-Guided Face Image Generation and Manipulationcited by 25
  23. 2022AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 26; 2025: 8; 2026: 3.

60 papers.

Sort:

By area:

  1. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  2. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  3. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  4. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  5. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  6. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  7. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  8. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  9. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  10. 2023DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generationcited by 75
  11. 2023CoSeR: Bridging Image and Language for Cognitive Super-Resolutioncited by 66
  12. 2024OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Modelscited by 64
38 more
  1. 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63
  2. 2023ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulationcited by 59
  3. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  4. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  5. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Imagescited by 42
  6. 2023DreamTuner: Single Image is Enough for Subject-Driven Generationcited by 41
  7. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  8. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  9. 2024IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Modelscited by 31
  10. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  11. 2023DreamIdentity: Improved Editability for Efficient Face-identity Preserved Image Generationcited by 26
  12. 2023LeftRefill: Filling Right Canvas based on Left Reference through Generalized Text-to-Image Diffusion Modelcited by 25
  13. 2023StoryGPT-V: Large Language Models as Consistent Story Visualizerscited by 25
  14. 2023ViCo: Plug-and-play Visual Condition for Personalized Text-to-image Generationcited by 23
  15. 2024Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Modelscited by 22
  16. 2025Personalized Image Generation with Deep Generative Models: A Decade Surveycited by 20
  17. 2024ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidancecited by 18
  18. 2024ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extractioncited by 16
  19. 2024ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generationcited by 16
  20. 2023Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generationcited by 15
  21. 2024EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Promptscited by 14
  22. 2023Visual Chain-of-Thought Diffusion Modelscited by 12
  23. 2024Correcting Diffusion-Based Perceptual Image Compression with Privileged End-to-End Decodercited by 11
  24. 2023SingleInsert: Inserting New Concepts from a Single Image into Text-to-Image Models for Flexible Editingcited by 10
  25. 2023Language-driven Scene Synthesis using Multi-conditional Diffusion Modelcited by 10
  26. 2023Cross Initialization for Personalized Text-to-Image Generationcited by 10
  27. 2024AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generationcited by 10
  28. 2024CusConcept: Customized Visual Concept Decomposition with Diffusion Modelscited by 9
  29. 2024StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customizationcited by 9
  30. 2023Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolutioncited by 7
  31. 2023Learning Profitable NFT Image Diffusions via Multiple Visual-Policy Guided Reinforcement Learningcited by 7
  32. 2024MetaDesigner: Advancing Artistic Typography Through AI-Driven, User-Centric, and Multilingual WordArt Synthesiscited by 6
  33. 2025Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generationcited by 6
  34. 2024DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generationcited by 5
  35. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
  36. 2024DreamBlend: Advancing Personalized Fine-tuning of Text-to-Image Diffusion Modelscited by 4
  37. 2025Growing Visual Generative Capacity for Pre-Trained MLLMscited by 4
  38. 2024ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Modelscited by 2

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  2. 2024Dual Diffusion for Unified Image Generation and Understanding
  3. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
  4. 2022Unified Discrete Diffusion for Simultaneous Vision-Language Generation
  5. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
  6. 2023Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net
  7. 2026MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation
  8. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models