mlsanityData to 7 Oct 2026

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

arXiv 2109.07306 ↗First posted 15 Sep 2021cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Compared to monolingual models, cross-lingual models usually require a more expressive vocabulary to represent all languages adequately. We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity. To this end, we propose an algorithm VoCap to determine the desired vocabulary capacity of each language. However, increasing the vocabulary size significantly slows down the pre-training speed. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax. Our experiments show that the multilingual vocabulary learned with VoCap benefits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mitigates the side-effects of increasing the vocabulary size while achieving comparable performance and faster pre-training speed. The code and the pretrained multilingual vocabularies are available at https://github.com/bozheng-hit/VoCapXLM.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

33 of 43 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
21 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  5. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  6. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  8. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  9. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  10. 2015Character-Aware Neural Language Modelscited by 129
  11. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  12. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  13. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  14. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  15. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  16. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  17. 2021MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairscited by 41
  18. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  19. 2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
  20. 2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24
  21. 2015Strategies for Training Large Vocabulary Neural Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 8; 2025: 3; 2026: 2.

22 papers.

Sort:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  3. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  4. 2022Language Modelling with Pixelscited by 53
  5. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  6. 2024Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddingscited by 20
  7. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  8. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 19
  9. 2025Scaling Embedding Layers in Language Modelscited by 19
  10. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  11. 2022Lessons learned from the evaluation of Spanish Language Modelscited by 8
  12. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
10 more
  1. 2024Structure-informed Positional Encoding for Music Generationcited by 5
  2. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
  3. 2026The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penaltycited by 4
  4. 2025Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processingcited by 3
  5. 2023HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understandingcited by 2
  6. 2023Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Searchcited by 2
  7. 2025Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiencycited by 2
  8. 2024Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!cited by 1
  9. 2024Towards Human-Like Machine Comprehension: Few-Shot Relational Learning in Visually-Rich Documentscited by 0
  10. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0