mlsanityData to 14 Oct 2026

Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation

arXiv 2103.10531 ↗First posted 18 Mar 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Successful methods for unsupervised neural machine translation (UNMT) employ crosslingual pretraining via self-supervision, often in the form of a masked language modeling or a sequence generation task, which requires the model to align the lexical- and high-level representations of the two languages. While cross-lingual pretraining works for similar languages with abundant corpora, it performs poorly in low-resource and distant languages. Previous research has shown that this is because the representations are not sufficiently aligned. In this paper, we enhance the bilingual masked language model pretraining with lexical-level information by using type-level cross-lingual subword embeddings. Empirical results demonstrate improved performance both on UNMT (up to 4.5 BLEU) and bilingual lexicon induction using our method compared to a UNMT baseline.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

23 of 29 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  6. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  7. 2017Word Translation Without Parallel Datacited by 487
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  9. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  10. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  11. 2016Using the Output Embedding to Improve Language Modelscited by 272
  12. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
11 more
  1. 2017Unsupervised Neural Machine Translationcited by 149
  2. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 146
  3. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 143
  4. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  5. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  6. 2018Unsupervised Statistical Machine Translationcited by 51
  7. 2020When Does Unsupervised Machine Translation Work?cited by 38
  8. 2019An Effective Approach to Unsupervised Machine Translationcited by 37
  9. 2020Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMTcited by 18
  10. 2014BilBOWA: Fast Bilingual Distributed Representations without Word Alignmentscited by 16
  11. 2019Explicit Cross-lingual Pre-training for Unsupervised Machine Translationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 4; 2024: 3; 2025: 0; 2026: 1.

14 papers.

Sort:

  1. 2021Survey of Low-Resource Machine Translationcited by 92
  2. 2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
  3. 2022Domain-Specific Text Generation for Machine Translationcited by 12
  4. 2024Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Studycited by 5
  5. 2024Self-Augmented In-Context Learning for Unsupervised Word Translationcited by 4
  6. 2021Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Studycited by 3
  7. 2022Cost-Effective Training in Low-Resource Neural Machine Translationcited by 2
  8. 2023On Bilingual Lexicon Induction with Large Language Modelscited by 2
  9. 2024POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translationcited by 2
  10. 2022Robust Unsupervised Cross-Lingual Word Embedding using Domain Flow Interpolationcited by 1
  11. 2023Investigating Lexical Sharing in Multilingual Machine Translation for Indian Languagescited by 1
  12. 2023On the Copying Problem of Unsupervised NMT: A Training Schedule with a Language Discriminator Losscited by 1
2 more
  1. 2023Denoising-based UNMT is more robust to word-order divergence than MASS-based UNMTcited by 0
  2. 2026Lius: Translation Model Based Instructional Lingustic Using Continual Instruction Tuning In Kupang Malaycited by 0