mlsanityData to 7 Oct 2026

Subword Mapping and Anchoring across Languages

arXiv 2109.04556 ↗First posted 9 Sep 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

State-of-the-art multilingual systems rely on shared vocabularies that sufficiently cover all considered languages. To this end, a simple and frequently used approach makes use of subword vocabularies constructed jointly over several languages. We hypothesize that such vocabularies are suboptimal due to false positives (identical subwords with different meanings across languages) and false negatives (different subwords with similar meanings). To address these issues, we propose Subword Mapping and Anchoring across Languages (SMALA), a method to construct bilingual subword vocabularies. SMALA extracts subword alignments using an unsupervised state-of-the-art mapping technique and uses them to create cross-lingual anchors based on subword similarities. We demonstrate the benefits of SMALA for cross-lingual natural language inference (XNLI), where it improves zero-shot transfer to an unseen language without task-specific data, but only by sharing subword embeddings. Moreover, in neural machine translation, we show that joint subword vocabularies obtained with SMALA lead to higher BLEU scores on sentences that contain many false positives and false negatives.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

48 of 62 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
36 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2019How multilingual is Multilingual BERT?cited by 642
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  5. 2017Word Translation Without Parallel Datacited by 480
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  7. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  8. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  9. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  10. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  11. 2019Massively Multilingual Neural Machine Translationcited by 246
  12. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  13. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  14. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  15. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  16. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  17. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  18. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  19. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  20. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  21. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  22. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 69
  23. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  24. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  25. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  26. 2021Multi-view Subword Regularizationcited by 40
  27. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  28. 2019Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spacescited by 35
  29. 2019Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Frameworkcited by 26
  30. 2019Analyzing the Limitations of Cross-lingual Word Embedding Mappingscited by 18
  31. 2020Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMTcited by 18
  32. 2016Learning Crosslingual Word Embeddings without Bilingual Corporacited by 12
  33. 2020On Romanization for Model Transfer Between Scripts in Neural Machine Translationcited by 9
  34. 2020Beyond Offline Mapping: Learning Cross Lingual Word Embeddings through Context Anchoringcited by 8
  35. 2018Integrating Weakly Supervised Word Sense Disambiguation into Neural Machine Translationcited by 7
  36. 2020Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 1; 2024: 3; 2025: 0; 2026: 3.

12 papers.

  1. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  2. 2022Language Modelling with Pixelscited by 53
  3. 2022Language-Family Adapters for Low-Resource Multilingual Neural Machine Translationcited by 14
  4. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 14
  5. 2024Multilingual Large Language Models and Curse of Multilingualitycited by 13
  6. 2022Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studiescited by 9
  7. 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systemscited by 8
  8. 2023Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMTcited by 4
  9. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
  10. 2022Robust Unsupervised Cross-Lingual Word Embedding using Domain Flow Interpolationcited by 1
  11. 2026Effective vocabulary expansion of multilingual language models for extremely low-resource languagescited by 0
  12. 2026Tokenizing Crosslingual Homographscited by 0