mlsanityData to 14 Oct 2026

Improving Multilingual Models with Language-Clustered Vocabularies

arXiv 2010.12777 ↗First posted 24 Oct 2020cs.CL cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

State-of-the-art multilingual models depend on vocabularies that cover all of the languages the model will expect to see at inference time, but the standard methods for generating those vocabularies are not ideal for massively multilingual applications. In this work, we introduce a novel procedure for multilingual vocabulary generation that combines the separately trained vocabularies of several automatically derived language clusters, thus balancing the trade-off between cross-lingual subword sharing and language-specific vocabularies. Our experiments show improvements across languages on key multilingual benchmark tasks TyDi QA (+2.9 F1), XNLI (+2.1%), and WikiAnn NER (+2.8 F1) and factor of 8 reduction in out-of-vocabulary rate, all without increasing the size of the model or data.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 10; 2023: 11; 2024: 11; 2025: 6; 2026: 4.

53 papers.

Sort:

By area:

  1. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  2. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  3. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  4. 2021MuRIL: Multilingual Representations for Indian Languagescited by 179
  5. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  6. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  7. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  8. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  9. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  10. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  11. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  12. 2022Language Modelling with Pixelscited by 55
38 more
  1. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  2. 2021Multi-view Subword Regularizationcited by 40
  3. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  4. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  5. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  6. 2022Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languagescited by 25
  7. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  8. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  9. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  10. 2022Text Generation with Text-Editing Modelscited by 21
  11. 2022How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?cited by 20
  12. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 20
  13. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  14. 2023Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lenscited by 15
  15. 2022Compositional Evaluation on Japanese Textual Entailment and Similaritycited by 14
  16. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
  17. 2022A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learningcited by 12
  18. 2023How Vocabulary Sharing Facilitates Multilingualism in LLaMA?cited by 12
  19. 2022Incorporating Context into Subword Vocabulariescited by 11
  20. 2024SUTRA: Scalable Multilingual Language Model Architecturecited by 11
  21. 2025False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Modelscited by 11
  22. 2023CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Codecited by 9
  23. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  24. 2021How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?cited by 7
  25. 2024Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmarkcited by 5
  26. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 5
  27. 2025Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languagescited by 5
  28. 2021Discovering Representation Sprachbund For Multilingual Pre-Trainingcited by 4
  29. 2024A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translationcited by 3
  30. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impactcited by 3
  31. 2025Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processingcited by 3
  32. 2022Cross-Lingual Knowledge Transfer for Clinical Phenotypingcited by 2
  33. 2024NLP-enabled Trajectory Map-matching in Urban Road Networks using a Transformer-based Encoder-decodercited by 2
  34. 2025Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insightscited by 2
  35. 2025SpeLLM: Character-Level Multi-Head Decodingcited by 2
  36. 2022Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERTcited by 1
  37. 2024Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!cited by 1
  38. 2026Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Modelscited by 1

50 most cited of 53 All 53 (JSON)