mlsanityData to 7 Oct 2026

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost

arXiv 2412.01271 ↗First posted 2 Dec 2024cs.CL cs.AIcited by 3

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages.Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 45 of its 54 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

45 of 54 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding14 May
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment8 Mar
  3. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation27 Feb
  4. 2023PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion27 Dec
  5. 2023Kandinsky 3.0 Technical Report6 Dec
  6. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation28 Nov
  7. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Model19 Aug
  8. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  9. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation17 Mar
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
1 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
20 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  8. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  9. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  10. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  11. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  12. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 242
  13. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  14. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  15. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  16. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  17. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  18. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  19. 2022DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generationcited by 30
  20. 2024Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Supportcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representationcited by 3
  2. 2025Un-Doubling Diffusion: LLM-guided Disambiguation of Homonym Duplicationcited by 1
  3. 2026StaTS: Spectral Trajectory Schedule Learning for Adaptive Time Series Forecasting with Frequency Guided Denoisercited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
  2. 2024Bridging Different Language Models and Generative Vision Models for Text-to-Image Generation
  3. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model
  4. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
  5. 2025TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis
  6. 2025Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis
  7. 2024Text Image Generation for Low-Resource Languages with Dual Translation Learning
  8. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs