mlsanityData to 14 Oct 2026

Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution

arXiv 2305.17325 ↗First posted 27 May 2023cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Zero-shot cross-lingual transfer is when a multilingual model is trained to perform a task in one language and then is applied to another language. Although the zero-shot cross-lingual transfer approach has achieved success in various classification tasks, its performance on natural language generation tasks falls short in quality and sometimes outputs an incorrect language. In our study, we show that the fine-tuning process learns language invariant representations, which is beneficial for classification tasks but harmful for generation tasks. Motivated by this, we propose a simple method to regularize the model from learning language invariant representations and a method to select model checkpoints without a development set in the target language, both resulting in better generation quality. Experiments on three semantically diverse generation tasks show that our method reduces the accidental translation problem by 68% and improves the ROUGE-L score by 1.5 on average.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

48 of 71 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
36 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  2. 2019How multilingual is Multilingual BERT?cited by 645
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  5. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  6. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  7. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  8. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  9. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  10. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 181
  11. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  12. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  13. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  14. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  15. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 96
  16. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  17. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  18. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  19. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  20. 2021Revisiting the Primacy of English in Zero-shot Cross-lingual Transfercited by 47
  21. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  22. 2018Neural Cross-Lingual Named Entity Recognition with Minimal Resourcescited by 44
  23. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  24. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  25. 2019Is Multilingual BERT Fluent in Language Generation?cited by 28
  26. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  27. 2021ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generationcited by 23
  28. 2021MTG: A Benchmark Suite for Multilingual Text Generationcited by 22
  29. 2022Transforming Sequence Tagging Into A Seq2Seq Taskcited by 15
  30. 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
  31. 2021Evaluating the Values of Sources in Transfer Learningcited by 11
  32. 2020Model Selection for Cross-Lingual Transfercited by 9
  33. 2022Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfercited by 7
  34. 2016Multi-Task Cross-Lingual Sequence Tagging from Scratchcited by 6
  35. 2022Meta-XNLG: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generationcited by 4
  36. 2022Zero-shot Cross-lingual Transfer is Under-specified Optimizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 3; 2026: 2.

22 papers.

Sort:

By area:

  1. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  2. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  3. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  4. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  5. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  6. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  7. 2024mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?cited by 21
  8. 2024Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative taskscited by 11
  9. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11
  10. 2025The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failurecited by 10
  11. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  12. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
10 more
  1. 2024Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysiscited by 7
  2. 2024Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasetscited by 6
  3. 2024Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysiscited by 4
  4. 2024Cross-lingual Transfer for Automatic Question Generation by Learning Interrogative Structures in Target Languagescited by 1
  5. 2023Empirical study of pretrained multilingual language models for zero-shot cross-lingual knowledge transfer in generationcited by 0
  6. 2023CUNI Submission to MRL 2023 Shared Task on Multi-lingual Multi-task Information Retrievalcited by 0
  7. 2024Language-Independent Representations Improve Zero-Shot Summarizationcited by 0
  8. 2024GNOME: Generating Negotiations through Open-Domain Mapping of Exchangescited by 0
  9. 2026Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignmentcited by 0
  10. 2026Language-Aware Token Boosting: LLM Language Confusion Reduction Without Tuningcited by 0