mlsanityData to 7 Oct 2026

Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivots

arXiv 2305.11540 ↗First posted 19 May 2023cs.CV cs.CLcited by 4

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models have made impressive progress in text-to-image synthesis. However, training such large-scale models (e.g. Stable Diffusion), from scratch requires high computational costs and massive high-quality text-image pairs, which becomes unaffordable in other languages. To handle this challenge, we propose IAP, a simple but effective method to transfer English Stable Diffusion into Chinese. IAP optimizes only a separate Chinese text encoder with all other parameters fixed to align Chinese semantics space to the English one in CLIP. To achieve this, we innovatively treat images as pivots and minimize the distance of attentive features produced from cross-attention between images and each language respectively. In this way, IAP establishes connections of Chinese, English and visual semantics in CLIP's embedding space efficiently, advancing the quality of the generated image with direct Chinese prompts. Experimental results show that our method outperforms several strong Chinese diffusion models with only 5%~10% training data.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 35 of its 48 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

35 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
20 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  3. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  4. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  5. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  6. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  7. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  8. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  9. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  10. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  11. 2019Controllable Text-to-Image Generationcited by 187
  12. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  13. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  14. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  15. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  16. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  17. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  18. 2021ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generationcited by 41
  19. 2019Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translationcited by 23
  20. 2022Large-scale Bilingual Language-Image Contrastive Learningcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  2. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  3. 2023PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloudcited by 5
  4. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generationcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding
  2. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Model
  3. 2024Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support
  4. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model
  5. 2025CTA-Flux: Integrating Chinese Cultural Semantics into High-Quality English Text-to-Image Communities
  6. 2024MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
  7. 2023Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
  8. 2022Diffusion-based Image Translation using Disentangled Style and Content Representation