mlsanityData to 14 Oct 2026

Translation-Enhanced Multilingual Text-to-Image Generation

arXiv 2305.19216 ↗First posted 30 May 2023cs.CL cs.AI cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated image-caption data in other languages; in the long run, this might widen inequitable access to TTI technology. In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems. We provide two key contributions. 1) Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: Translate Train, Translate Test, and Zero-Shot Transfer. 2) We propose Ensemble Adapter (EnsAd), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance. Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of translation-enhanced mTTI systems and also validate the benefits of the proposed EnsAd which derives consistent gains across all datasets. Further investigations on model variants, ablation studies, and qualitative analyses provide additional insights on the inner workings of the proposed mTTI approaches.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

54 of 65 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
38 more
  1. 2016Improved Techniques for Training GANscited by 2,970
  2. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  5. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2019Green AIcited by 627
  13. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  14. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  15. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  16. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  17. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 332
  18. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  19. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  20. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  21. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  22. 2018A Note on the Inception Scorecited by 226
  23. 2019Connecting Vision and Language with Localized Narrativescited by 226
  24. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
  25. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 162
  26. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 128
  27. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 123
  28. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  29. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  30. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  31. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  32. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  33. 2019How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptionscited by 48
  34. 2017Fluency-Guided Cross-Lingual Image Captioningcited by 33
  35. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  36. 2018Marian: Cost-effective High-Quality Neural Machine Translation in C++cited by 22
  37. 2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
  38. 2021Modelling Latent Translations for Cross-Lingual Transfercited by 16

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 1; 2026: 3.

9 papers.

  1. 2024CALRec: Contrastive Alignment of Generative LLMs for Sequential Recommendationcited by 24
  2. 2025Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generationcited by 11
  3. 2023On the Cultural Gap in Text-to-Image Generationcited by 10
  4. 2023PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generationcited by 5
  5. 2024Navigating Text-to-Image Generative Bias across Indic Languagescited by 5
  6. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
  7. 2026Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Textcited by 0
  8. 2026LogoDiffuser: Training-Free Multilingual Logo Generation and Stylization via Letter-Aware Attention Controlcited by 0
  9. 2026Presence Is Not Faithfulness: Figurative Vehicle Intrusion in Text-to-Image Generationcited by 0