mlsanityData to 7 Oct 2026

Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages

arXiv 2109.05772 ↗First posted 13 Sep 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements. Typically, subword tokenization algorithms such as byte pair encoding and WordPiece are used. In this work, we investigate the compatibility of tokenizations for multilingual static and contextualized embedding spaces and propose a measure that reflects the compatibility of tokenizations across languages. Our goal is to prevent incompatible tokenizations, e.g., "wine" (word-level) in English vs. "v i n" (character-level) in French, which make it hard to learn good multilingual semantic representations. We show that our compatibility measure allows the system designer to create vocabularies across languages that are compatible -- a desideratum that so far has been neglected in multilingual models.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

28 of 40 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
16 more
  1. 2019How multilingual is Multilingual BERT?cited by 642
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  4. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  5. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  6. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 189
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  8. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  9. 2019Emerging Cross-lingual Structure in Pretrained Language Modelscited by 180
  10. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  11. 2020Zero-Shot Cross-Lingual Transfer with Meta Learningcited by 66
  12. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
  13. 2020A Call for More Rigor in Unsupervised Cross-lingual Learningcited by 37
  14. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
  15. 2019Should All Cross-Lingual Embeddings Speak English?cited by 14
  16. 2020Subword Sampling for Low Resource Word Alignmentcited by 4

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 2; 2024: 1.

8 papers.

  1. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  2. 2022Language Modelling with Pixelscited by 53
  3. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  4. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  5. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12
  6. 2022Incorporating Context into Subword Vocabulariescited by 11
  7. 2024A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translationcited by 3
  8. 2023Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Modelscited by 1