mlsanityData to 7 Oct 2026

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

arXiv 2301.10472 ↗First posted 25 Jan 2023cs.CL cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This textitvocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R. In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), to named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 11.2% and 5.8% absolute on MasakhaNER and Americas NLI, respectively.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 25 Jan 2023.

Builds on

28 of 37 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
16 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  2. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  3. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  4. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  5. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  7. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  8. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  9. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  10. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  11. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  12. 2016Efficient softmax approximation for GPUscited by 67
  13. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  14. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  15. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 27
  16. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 32; 2025: 25; 2026: 11.

88 papers.

Sort:

By area:

  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  3. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  4. 2024Tokenization Is More Than Compressioncited by 87
  5. 2024Getting the most out of your tokenizer for pre-training and domain adaptationcited by 65
  6. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  7. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 61
  8. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 59
  9. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  10. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  11. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  12. 2024Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentationcited by 41
38 more
  1. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  2. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  3. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  4. 2024Greed is All You Need: An Evaluation of Tokenizer Inference Methodscited by 34
  5. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  6. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  7. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  8. 2024BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Trainingcited by 30
  9. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 23
  10. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  11. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  12. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  13. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 19
  14. 2025Scaling Embedding Layers in Language Modelscited by 19
  15. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 15
  16. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  17. 2023Do large language models solve verbal analogies like children do?cited by 13
  18. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  19. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 11
  20. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 11
  21. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  22. 2024Targeted Multilingual Adaptation for Low-resource Language Familiescited by 9
  23. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  24. 2023OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratchcited by 7
  25. 2023'UFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolutioncited by 7
  26. 2024PersianMind: A Cross-Lingual Persian-English Large Language Modelcited by 7
  27. 2024English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Promptscited by 7
  28. 2024KazQAD: Kazakh Open-Domain Question Answering Datasetcited by 7
  29. 2024TookaBERT: A Step Forward for Persian NLUcited by 7
  30. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  31. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  32. 2024How Multilingual Are Large Language Models Fine-Tuned for Translation?cited by 6
  33. 2024How Transliterations Improve Crosslingual Alignmentcited by 6
  34. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  35. 2024Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmarkcited by 5
  36. 2025Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?cited by 5
  37. 2025Model-Aware Tokenizer Transfercited by 5
  38. 2023A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Modelscited by 4

50 most cited of 88 All 88 (JSON)