mlsanityData to 14 Oct 2026

MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation

arXiv 2305.15296 ↗First posted 24 May 2023cs.CV cs.AI cs.LGcited by 15

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in natural language, with the model producing faithful interpretations of text prompts. However, expressing complex or nuanced ideas in text alone can be difficult. To ease image generation, we propose MultiFusion that allows one to express complex and nuanced concepts with arbitrarily interleaved inputs of multiple modalities and languages. MutliFusion leverages pre-trained models and aligns them for integration into a cohesive system, thereby avoiding the need for extensive training from scratch. Our experimental results demonstrate the efficient transfer of capabilities from individual modules to the downstream model. Specifically, the fusion of all independent components allows the image generation module to utilize multilingual, interleaved multimodal inputs despite being trained solely on monomodal data in a single language.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 48 of its 57 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

48 of 57 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation17 Mar
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  5. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  6. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  8. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts27 Oct
  9. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models26 Jul
  10. 2022Blended Latent Diffusion6 Jun
  11. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
2 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  2. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
22 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  13. 2023Universal Guidance for Diffusion Modelscited by 437
  14. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  15. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  16. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  17. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 151
  18. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  19. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  20. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
  21. 2023AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulationcited by 31
  22. 2022ILLUME: Rationalizing Vision-Language Models through Human Interactionscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 7; 2025: 3.

15 papers.

Sort:

By area:

  1. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  2. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  3. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  4. 2023Quality-Diversity through AI Feedbackcited by 40
  5. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  6. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Modelscited by 20
  7. 2023Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the World's Ugliness?cited by 16
  8. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  9. 2023Can pre-trained models assist in dataset distillation?cited by 12
  10. 2023Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimizationcited by 10
  11. 2025Towards an Explainable Comparison and Alignment of Feature Embeddingscited by 10
  12. 2025UniFusion: Vision-Language Model as Unified Encoder in Image Generationcited by 8
3 more
  1. 2024Position: Towards Implicit Prompt For Text-To-Image Modelscited by 1
  2. 2024Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variationscited by 0
  3. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
  2. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion
  3. 2024MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
  4. 2024Dual Diffusion for Unified Image Generation and Understanding
  5. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Models
  6. 2023Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else
  7. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generation
  8. 2026UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation