mlsanityData to 7 Oct 2026

Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages

arXiv 2408.02290 ↗First posted 5 Aug 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space. Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and easily adaptable to new languages. In this work, we test this hypothesis by zero-shot translating from unseen languages. To deal with unknown vocabularies from unknown languages we propose a setup where we decouple learning of vocabulary and syntax, i.e. for each language we learn word representations in a separate step (using cross-lingual word embeddings), and then train to translate while keeping those word representations frozen. We demonstrate that this setup enables zero-shot translation from entirely unseen languages. Zero-shot translating with a model trained on Germanic and Romance languages we achieve scores of 42.6 BLEU for Portuguese-English and 20.7 BLEU for Russian-English on TED domain. We explore how this zero-shot translation capability develops with varying number of languages seen by the encoder. Lastly, we explore the effectiveness of our decoupled learning strategy for unsupervised machine translation. By exploiting our model's zero-shot translation capability for iterative back-translation we attain near parity with a supervised setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

45 of 62 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  11. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
33 more
  1. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  2. 2017Word Translation Without Parallel Datacited by 480
  3. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  4. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  5. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  6. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  7. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  8. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  9. 2017Unsupervised Neural Machine Translationcited by 148
  10. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  11. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  12. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  13. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  14. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  15. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  16. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
  17. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 69
  18. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  19. 2018Unsupervised Statistical Machine Translationcited by 51
  20. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  21. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  22. 2020When Does Unsupervised Machine Translation Work?cited by 38
  23. 2019An Effective Approach to Unsupervised Machine Translationcited by 37
  24. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  25. 2020When and Why is Unsupervised Neural Machine Translation Useless?cited by 34
  26. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  27. 2019Do We Really Need Fully Unsupervised Cross-Lingual Embeddings?cited by 31
  28. 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
  29. 2019Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabulariescited by 27
  30. 2018Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputscited by 26
  31. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  32. 2019From Bilingual to Multilingual Neural Machine Translation by Incremental Trainingcited by 14
  33. 2022Multilingual Bidirectional Unsupervised Translation Through Multilingual Finetuning and Back-Translationcited by 3

Built on by

Not cited yet.