mlsanityData to 7 Oct 2026

Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

arXiv 2410.09644 ↗First posted 12 Oct 2024cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Vocabulary adaptation, which integrates new vocabulary into pre-trained language models, enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model's weights fixed. VocADT offers a flexible and scalable solution without depending on external resources or language constraints. Across 11 languages-with diverse scripts, resource availability, and fragmentation-we demonstrate that VocADT outperforms the original Mistral model and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 58 of its 76 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

58 of 76 references.

References · 58

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  12. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
38 more
  1. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  2. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  3. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  4. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  5. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  6. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  7. 2019Massively Multilingual Neural Machine Translationcited by 246
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  9. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  10. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  11. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  13. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  14. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  15. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  16. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  17. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  18. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  19. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
  20. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  21. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  22. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
  23. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  24. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  25. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  26. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  27. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  28. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  29. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  30. 2024Zero-Shot Tokenizer Transfercited by 30
  31. 2017Regularization techniques for fine-tuning in neural machine translationcited by 27
  32. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  33. 2019Low-Resource Corpus Filtering using Multilingual Sentence Embeddingscited by 25
  34. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  35. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  36. 2024An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Modelscited by 20
  37. 2018Compact Personalized Models for Neural Machine Translationcited by 19
  38. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 4.

9 papers.

  1. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  2. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  3. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  4. 2025Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionariescited by 2
  5. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  6. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1
  7. 2026Understanding Secret Leakage Risks in Code LLMs: A Tokenization Perspectivecited by 0
  8. 2026HiMed: Incentivizing Hindi Reasoning in Medical LLMscited by 0
  9. 2026Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustnesscited by 0