# AltDiffusion: A Multilingual Text-to-Image Diffusion Model

arXiv: https://arxiv.org/abs/2308.09991 · HTML: https://mlsanity.com/paper/2308.09991/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 19 Aug 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [PP-OCR: A Practical Ultra Lightweight OCR System](https://mlsanity.com/paper/2009.09941/) (2020)
- [ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts](https://mlsanity.com/paper/2210.15257/) (2022)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence](https://mlsanity.com/paper/2209.02970/) (2022)
- [ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation](https://mlsanity.com/paper/2112.15283/) (2021)
- [Towards Zero-shot Cross-lingual Image Retrieval](https://mlsanity.com/paper/2012.05107/) (2020)

## Built on by (most cited first, 30 of 30)

- [Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](https://mlsanity.com/paper/2405.08748/) (2024)
- [Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models](https://mlsanity.com/paper/2310.01929/) (2023)
- [Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You](https://mlsanity.com/paper/2401.16092/) (2024)
- [Neodragon: Mobile Video Generation using Diffusion Transformer](https://mlsanity.com/paper/2511.06055/) (2025)
- [Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support](https://mlsanity.com/paper/2401.14688/) (2024)
- [An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation](https://mlsanity.com/paper/2405.12914/) (2024)
- [Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generation](https://mlsanity.com/paper/2511.17282/) (2025)
- [TraceRouter: Robust Safety for Large Foundation Models via Path-Level Intervention](https://mlsanity.com/paper/2601.21900/) (2026)
- [ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2](https://mlsanity.com/paper/2407.19832/) (2024)
- [DreamID: High-Fidelity and Fast diffusion-based Face Swapping via Triplet ID Group Learning](https://mlsanity.com/paper/2504.14509/) (2025)
- [MirrorVerse: Pushing Diffusion Models to Realistically Reflect the World](https://mlsanity.com/paper/2504.15397/) (2025)
- [UniFusion: Vision-Language Model as Unified Encoder in Image Generation](https://mlsanity.com/paper/2510.12789/) (2025)
- [When Cultures Meet: Multicultural Text-to-Image Generation](https://mlsanity.com/paper/2502.15972/) (2025)
- [OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure](https://mlsanity.com/paper/2603.11493/) (2026)
- [CGI-DM: Digital Copyright Authentication for Diffusion Models via Contrasting Gradient Inversion](https://mlsanity.com/paper/2403.11162/) (2024)
- [PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation](https://mlsanity.com/paper/2311.17086/) (2023)
- [FlexCache: Flexible Approximate Cache System for Video Diffusion](https://mlsanity.com/paper/2501.04012/) (2024)
- [PIXELS: Progressive Image Xemplar-based Editing with Latent Surgery](https://mlsanity.com/paper/2501.09826/) (2025)
- [iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Design](https://mlsanity.com/paper/2312.04326/) (2023)
- [PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion](https://mlsanity.com/paper/2312.16486/) (2023)
- [Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering](https://mlsanity.com/paper/2401.07510/) (2024)
- [MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost](https://mlsanity.com/paper/2412.01271/) (2024)
- [LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation](https://mlsanity.com/paper/2502.18302/) (2025)
- [Evaluation of Cultural Competence of Vision-Language Models](https://mlsanity.com/paper/2505.22793/) (2025)
- [Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models](https://mlsanity.com/paper/2410.03039/) (2024)
- [Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text](https://mlsanity.com/paper/2601.10096/) (2026)
- [SoS: Analysis of Surface over Semantics in Multilingual Text-To-Image Generation](https://mlsanity.com/paper/2601.16803/) (2026)
- [LogoDiffuser: Training-Free Multilingual Logo Generation and Stylization via Letter-Aware Attention Control](https://mlsanity.com/paper/2603.09759/) (2026)
- [SciDraw-6K: A Multilingual Scientific Illustration Dataset Generated by Google Gemini](https://mlsanity.com/paper/2604.17206/) (2026)
- [BAFIS: Dataset + Framework to assess occupational Bias and Human Preference in modern Text-to-image Models](https://mlsanity.com/paper/2606.20241/) (2026)

## Abstract (arXiv)

Large Text-to-Image(T2I) diffusion models have shown a remarkable capability to produce photorealistic and diverse images based on text inputs. However, existing works only support limited language input, e.g., English, Chinese, and Japanese, leaving users beyond these languages underserved and blocking the global expansion of T2I models. Therefore, this paper presents AltDiffusion, a novel multilingual T2I diffusion model that supports eighteen different languages. Specifically, we first train a multilingual text encoder based on the knowledge distillation. Then we plug it into a pretrained English-only diffusion model and train the model with a two-stage schema to enhance the multilingual capability, including concept alignment and quality improvement stage on a large-scale multilingual dataset. Furthermore, we introduce a new benchmark, which includes Multilingual-General-18(MG-18) and Multilingual-Cultural-18(MC-18) datasets, to evaluate the capabilities of T2I diffusion models for generating high-quality images and capturing culture-specific concepts in different languages. Experimental results on both MG-18 and MC-18 demonstrate that AltDiffusion outperforms current state-of-the-art T2I models, e.g., Stable Diffusion in multilingual understanding, especially with respect to culture-specific concepts, while still having comparable capability for generating high-quality images. All source code and checkpoints could be found in https://github.com/superhero-7/AltDiffuson.